summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* guest-diag: nmi_catch.sh - sysrq-l sampler that keeps nvidia kernel stacksArthur Hanson Rasmusson2026-09-28
| | | | | | | | | | The guest RM's expiring waits are kernel busy-polls and all of their functions are notrace, so ftrace/kprobes cannot name them. sysrq-l NMI backtraces can: this sampler triggers them and keeps only the Comm and [nvidia] frame lines per round, so the wait/teardown stacks survive dmesg ring wraps. Caught the adapter-shutdown scrubber stack (channelReadChannelMemdesc <- ceutilsUpdateProgress <- scrubberDestruct <- memmgrDestroyInternalChannels <- gpuStatePreUnload <- nv_shutdown_adapter).
* sema: clamp the pin-rewind at ring start - never wrap below zeroArthur Hanson Rasmusson2026-09-28
| | | | | | | The rewind used entries - (N - get) for get < N, which for an empty ring (get=0, put=0) jumped last_get to entries-8, so the pump read ring slots the producer had never reached and could execute stale/garbage entries. Replay only recently consumed slots, clamped at 0.
* sema: re-run recent ring entries when a translation is replacedArthur Hanson Rasmusson2026-09-28
| | | | | | | | | | | | | | | | | A pin injected after the pump consumed pushes that release through that VA resolved against the old (or bogus) translation, so the releases were compared as stale against the old page's values and dropped - a new client that maps fresh pages at the same VAs then waits forever. Rewind every channel's last_get by 8 ring entries on injection so the pump re-reads those pushes under the new translation; the release and fill application paths are forward-progress guarded and idempotent, so re-execution is safe. The handler runs in the service thread, which also runs the pump, so no pass can be in flight during the rewind. Diagnosed from cdt4: its tracker releases were consumed while the pin still pointed at the previous cdt's page (value 1), so payload <= cur and every release was dropped as stale; re-pinning afterwards could not resurrect them because the ring entries had already been consumed.
* sema: never execute work decoded from swept RAM; pin only tracker semsArthur Hanson Rasmusson2026-09-28
| | | | | | | | | | | | | | | | | | | The content sweep reads arbitrary guest RAM and ran the full push decoder on every page, so fabricated method streams in unrelated data executed fills/copies: one boot scanned 65M pages, 'decoded' 124k pushes and ran dozens of memsets from false positives - writes into live guest memory (a guest kernel crashed in nvidia_uvm right after such a boot). - Swept pages now only permit the bounded semaphore-release writes; fills and copies execute only for pushes the pump read through a trusted ring translation. - Pushes read for a compute (GR) channel apply releases only: their method space is not the copy engine's, so aliased encodings must never become memory writes. - sempages.py derives the pages to pin from the client's tracker objects (sem object + sem memory) instead of matching bare sem-object shapes, which pinned 140 pages where the client has ~53 sems; shape scan remains as a fallback when no tracker resolves.
* sema: GR ring autopin must not trust an early base or a pre-existing pinArthur Hanson Rasmusson2026-09-28
| | | | | | | | | | | | | | | | | | | | | | Two failure modes kept every GR channel dark despite the ring layout being known: - The CE region size was latched on the first autopin attempt, which can run before all CE channels are registered; a too-small base then points every GR candidate at a chunk whose entry 0 is zeros, and it stays wrong for the life of the boot. Recompute the base from the live CE ranks on every pass. - An existing pin for a ring VA (e.g. from a convention guess) made the autopin mark the rank done without validating the ring - a wrong USERD is then never corrected. Validate the candidate chunk (entry 0 decode + USERD put/get) first and only fall back to trusting an existing pin when the chunk does not validate; on success overwrite the pin. Also log one diagnostic line per GR rank when the chunk does not validate yet, so a silent autopin failure is visible in dmesg. Live: fresh boot, no manual ring pins - 20/20 GR channels auto-pinned (correct USERD 0x142000 + 0x3000*i) and ran entry 0 (get=1 sub=1) once the push pages were bridged.
* sem: run GR channels - accept CE methods on subchannel 0, discover GR ringsArthur Hanson Rasmusson2026-09-28
| | | | | | | | | | | | | | | | | | | | | - The CE semaphore/memset method groups also arrive on subchannel 0 (CUDA's push library binds the copy class on compute channels to carry the work-submission releases), so the stream decoder accepts both 0 and 4. - GR (compute) channel rings live in the same video-memory allocation as the CE rings, 0x3000 bytes per channel after them: ring (0x2000) at ce_end + 0x3000*rank, USERD at +0x2000. Discover and pin them once the CE ranks are known; verified against ring entry 0 (1 MB aligned push VA in the client window, sane length) and the USERD's put/get, so a layout change costs a failed pin, not a wrong one. Gated by sema_gr_rings (default on). - memprobe: require the sem-VA triple in the sem-object scanner (a bare qword in the window matches thousands of unrelated pointers once the ring/pushbuffer windows are part of the range). Live evidence: ring entries read through the guest's /proc/pid/mem decode to the GR channels' first pushes; pinning the ring (fb 0x140000+0x3000*rank), USERD (ring+0x2000) and first-push pages made all 20 GR channels execute their entry 0 for the first time (get=1 sub=1).
* sema-svc: apply releases promptly and decode compute-push releasesArthur Hanson Rasmusson2026-09-28
| | | | | | | | | | | | | | | | | | The content sweep now: - decodes a release whose method stream continues with any INC_METHOD, not only the CE LAUNCH_DMA group, so the releases at the head of a compute/GR channel's first push are applied too; - remembers releases whose target VA had no translation (deferred list) and retries them every service pass, so a VA->PA injection lands them immediately instead of after the next full sweep; - re-arms the fast sliced sweep for a few seconds after an injection; - treats a channel whose USERD/ring cannot be resolved as stalled, so the sweep keeps looking for the releases its unreadable pushes carry. tools: add per-cdt client-pin helper (sem pages + /dev/nvidia0 windows), sem-page scanner and memprobe fixes (short-read guard, --brief). Live-verified: injecting a client sem page applied 12 deferred releases within a second; the service kept applying (89 -> 202) while cdt ran.
* gsp_rm: advertise two GRCEs on TU104 (host reports 3 async CEs of 5); CE1 ↵Arthur Hanson Rasmusson2026-09-26
| | | | shares GR's runlist/PBDMA
* tools/guest-diag: decode engine-list, classlist and CE-cap RM control ↵Arthur Hanson Rasmusson2026-09-26
| | | | replies in the ioctl tracer
* kernel: add a channel-table dump param; tools: fix memprobe --max parsingArthur Hanson Rasmusson2026-09-26
|
* gsp_rm: cap event records between replies so the status ring cannot wrapArthur Hanson Rasmusson2026-09-26
|
* docs/uvm: describe the push-execution fallbacks and the semaphore-VA flap ↵Arthur Hanson Rasmusson2026-09-26
| | | | pitfall
* tools/guest-diag: keep bridge_sems replies under the guest-agent size limitArthur Hanson Rasmusson2026-09-26
|
* gsp_sema_service: apply a fast sliced RAM sweep when a channel's pushes stallArthur Hanson Rasmusson2026-09-26
|
* gsp_rm: make GSP_SET_SYSTEM_INFO/SET_REGISTRY fire-and-forget (no reply)Arthur Hanson Rasmusson2026-09-25
| | | | | | Guest driver was receiving SUCCESS replies as 'unexpected events' (0x48, 0x49). These async init RPCs are fire-and-forget per GSP-RM protocol; driver doesn't expect a reply on the status queue. CONTINUATION_RECORD still sends SUCCESS.
* gsp_rm: implement missing RM_CONTROL handlers for 570.144 guest driverArthur Hanson Rasmusson2026-09-25
| | | | | | | | | | | | | | | | - Added TU104_CTRL_INTERNAL_MEMSYS_L2_INVALIDATE_EVICT (0x20800a6c) - Added TU104_CTRL_INTERNAL_GR_GET_FECS_TRACE_HW_ENABLE (0x20800a38) - Added TU104_CTRL_SYSTEM_MAP_SHARED_DATA (0x730151) - Fixed async RPCs (72 GSP_SET_SYSTEM_INFO, 73 SET_REGISTRY, 71 CONTINUATION_RECORD) to send SUCCESS replies instead of silent consume - Added explicit handlers for RPC 47 (UNLOADING_GUEST_DRIVER) and 70 (UPDATE_BAR_PDE) with proper reply semantics Guest driver was failing at GSP-RM level with unknown RPCs 47, 70, 72, 73. Channels registered then immediately unregistered. Now they should stay registered and complete initialization.
* gsp_sema_service: add content rescue for unmapped push-buffer pool VAsArthur Hanson Rasmusson2026-09-25
| | | | | | | | | | - Add expected_sem_payload tracking to svc_channel for CE channels - Implement svc_rescue_push_page() to scan guest RAM for push with expected semaphore release when VA translation fails - Add svc_init_expected_payload() to initialize expected payload from pool - Add rate-limited logging of distinct stalled VAs (svc_log_stalled_va) - Advance expected_sem_payload on successful push decode - Pushes SEMA_MAP_MAX to 65536, SVC_PIN_SLOTS to 32768 for scale
* kernel: enlarge the pin table and its probe window; tools: bridge client ↵Arthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | | | | | mappings into the service The pin table became the primary VA->PA store once push-buffer regions and bridged client mappings started adding thousands of pages per boot; at 8192 slots with an 8-probe window, inserts were silently dropped and lookups missed at half load. 32768 slots and a 32-probe window keep the mapped set intact. tools/guest-diag/{guest_pmap_all.py,bridge_sems.sh} translate the CUDA client's /dev/nvidia* mappings through the guest's own pagemap and inject each present page into the service as a sysmem pin. With the client's semaphore pages pinned the service applies their releases into the page the client polls (verified: applied 67 -> 109, misses stop growing for those VAs), where before every write went to the vidmem shadow.
* kernel: resolve semaphores through trusted translations only; tools: dump ↵Arthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | | | | | | | the libcuda tracker wait The vidmem window convention turns any VA above the window base into a vidmem offset, but the CUDA client's GPU semaphores are sysmem objects mapped into its address space at those same VAs. A release applied through the convention therefore landed in the vidmem shadow instead of the page the client polls, and the completion was silently lost: the libcuda tracker it waits on never advanced. Semaphore reads and writes now use trusted translations only (pins, learned cache, discovered map, page walks), so an unresolved semaphore is counted as a miss and stays visible instead of corrupting a shadow. tools/guest-diag/trkinfo.py dumps the libcuda tracker the client spins on: the tracker object behind p+0x9410, the semaphore object it references, the u32 semaphore memory it polls (a /dev/nvidiactl mapping backed by guest RAM) and the VMA that memory lives in.
* kernel: self-configure CE rings and push buffers in the semaphore serviceArthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | | | | | | | | | The service pins the CE channel rings and push buffers without any guest-supplied addresses. Rings are derived from the channel rank among the live CE channels (ring at FB 0x80000 + 0x20000*rank, USERD at ring + 0x10000) and verified against the ring's first GPFIFO entry before the pin is stored. Push-buffer base pages are found by their first-push signature (SET_OBJECT on subchannel 0, SET_OBJECT on the compute subchannel, tracking-semaphore release at page offset 0x10 with payload 1); the release's in-page slot offset names the channel and the ring's first entry names the push-buffer VA, then the whole 1 MB region is pinned and marked hot. Pinning a pair also re-arms the page-directory discovery and feeds it the verified VA/PA pairs as samples, so the walk can translate the parts of the pool no base page describes. With rings and push buffers pinned the pump executes the submitted work (gpfifo done grows, semaphore releases apply, released kept in vidmem) and the guest kernel UVM waits complete.
* kernel: aperture-aware pinning and UVM CE channel USERD discoveryArthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | | | Debug injections could only teach sysmem translations, but the UVM CE channel rings and USERDs live in the frame buffer window, and the progress service had no candidate for their USERD (the allocation parameters carry an offset relative to a client pool, not the FB offset). Add an aperture field to the pinned translations and to the sema_map_add debug parameter (va,pa[,len[,aperture]]), and probe the observed UVM layout as well: the USERD sits one 0x10000 block after the ring for the data-path channels. With the rings pinned, the pump now resolves and decodes the CE rings (the entries point at 1 MB pushbuffer regions); only the pushbuffer VA->PA translation is still missing, so the entries are kept pending.
* tools/guest-diag: add UVM rm_mem kprobe dump for VA->PA ground truthArthur Hanson Rasmusson2026-09-24
| | | | | | | The probe dumps {type, CPU VA, GPU VAs, size} on every uvm_rm_mem_get_cpu_va call (and the first 64 bytes of the object's CPU mapping with nested fetchargs), which is how the host-side learned map can be checked against what the guest actually created.
* kernel: extend RAM scan to the full guest window and rescan hot push pagesArthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | | | The learned VA->PA map only covered guest physical addresses below 5 GB, but a 4 GB guest has RAM low plus a remapped window above the PCI hole (top around 6 GB). Structures that landed in the top gigabyte (for example the UVM semaphore pool) were never discovered, which made the semaphore translations flaky from boot to boot: releases then could not be applied and the guest's tracker waits never completed. Scan and accept translations over the full 6 GB window, and rescan the pages that have carried release groups in the rolling sweep (pushbuffer pools are recycled, so those pages hold the next releases). Also log the first distinct release misses and the sweep position so a stuck wait can be told apart from a translation gap.
* tools/guest-diag: add UVM wait diagnostics, tracker poke and VA->PA map injectorArthur Hanson Rasmusson2026-09-24
|
* kernel: completion-event hints and GSP-RM state introspectionArthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | - re-post the completion event for a bounded window after a channel completes work (notifier consumed or semaphore release applied), so a client that starts waiting after the edge-triggered wake-up still gets a wake-up; the OS event classes carry no source object, so the hint broadcasts to the registered events - add rmstate (dump msgq cursors, guest consumption position, event table) and rmpost (post an event record by index or to all) debug parameters, both executing in kthreads because the IOMMU read path only works with a NULL current->mm or the mapping owner's mm - report the underlying error codes from the event post path
* kernel: validate learned semaphore maps and post guest GPU eventsArthur Hanson Rasmusson2026-09-24
| | | | | | | | | | | | | | | - admit {GPU VA -> PA} pairs only when the preceding bytes form a valid client memory descriptor header (type 0/1, CPU+GPU mapped bits), prefer validated pairs over headerless candidates under first-wins dedup, and invalidate the translation cache on every rebuild; fixes stale identity entries shadowing live pool pages and lets the guest UVM channel bring-up complete on its own - capture OS event allocations (class 0x79) and post event records (function 0x1003) into the guest status queue when a channel's busy notification is consumed, with a broadcast fallback, serialized with replies via a queue lock - add BAR0 shadow introspection (dump_bar0/scan_bar0) and guest-side diagnostic scripts for the VM cycle
* kernel: track channels and execute host-class pushed workArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The guest's CUDA stack was stuck inside context creation because the work it submits is never executed by the emulated GPU. Two forms of pushed work were missing, and the completion path had grown far too slow to be useful: - Channel tracking: on a channel allocation parse the GPFIFO GPU VA and entry count, the USERD address/aperture and the instance block, hand them to the progress service and drop them again when the channel is freed. - Host-class (C46F) semaphore releases: pushes on a compute channel signal progress with a five-dword method run (SEM_ADDR_LO/HI, PAYLOAD_LO/HI, EXECUTE) rather than the copy-engine group the service already understood, so those releases were never applied. Decode both forms. - Emulated vidmem: channel blocks (USERD/instance) and some client semaphores live in the emulated frame buffer, whose VAs are a window over the BAR1 backing rather than guest RAM; access them through the right aperture. - Latency: the service's full-RAM sweep took seconds per pass, so guests' progress waits took whole seconds and the driver tore its own setup back down in the meantime. Sweep guest RAM in bounded, chunked windows on every pass, remember the pages pushes are actually found in and revisit those first, and keep the VA->PA map complete (dedup, sort, binary search) so semaphores resolve reliably. Also add a vidmem poke parameter for bringing up and debugging emulated frame-buffer state.
* docs: merge the legacy flat documents, add the CUDA deep-dive and link checksArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | | | | * Merge and remove the pre-reboot flat documents, preserving their still-useful material in the structured tree: debugging workflows and the EPT-livelock case study (05-operations/02), performance microbenchmarks and A/B methodology (05-operations/01), the version-upgrade runbook (05-operations/02 section 4, with corrected host/guest framing), kernel-code pitfalls (07-developers) and the OpenRM build/load order (03-architecture/06 section 7). The remaining legacy files were verified redundant and removed. * docs/README now records where each migrated topic lives. * Portability: the config YAML files are called out as design targets that no code reads yet, with what adding support actually requires today. * Getting started: the OpenRM step is marked optional (native vGPU-management workflows only; the emulated path uses the distro driver). * tools docs: rmtool/rmtop marked as planned, pointing at the implemented tools/rmq subset. * wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the canonical userspace/cuda analysis) and index update. * scripts/check-docs-links.py plus a CI step so relative links cannot rot. * STATE.md references to removed documents repointed.
* userspace/cuda: complete the analysis passArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Name recovery: name_candidates.py proposes the hand-written API-layer implementation for each exported wrapper by grouping version variants and intersecting their callees; 180 high-confidence proposals recorded in annotations/name-candidates.tsv, 29 of them body-verified and promoted to names.tsv with evidence in subsystems.tsv. * Engine records: identified the 0x14-byte per-engine descriptor, mapped its fields (kind, engine type, instance, class id, CE caps) and the existence callback, and documented how the missing async copy engine (COPY14) leaves the secondary record zeroed and blocks context creation (docs/05 section 3.7, public-safe). * Targeted type recovery: AnalyzeFunctions.java runs the parameter-ID analyzer over a curated list of 80 functions; 16 improved materially and the typed decompilations are kept under the version run tree. * Cross-version: cudatool.py fdematch measures how names transfer between patch versions (570.144 -> 570.211.01: exports 100%, verified internals ~70%, generated wrappers must be re-resolved by relocation, never by address arithmetic); documented in the upgrade playbook. * Corpus tooling: diff_corpus.py (version-to-version corpus diff) and subsystem_digest.py (per-subsystem call trees). * Deprecate the old per-version helpers whose address handling and analysis timeout produced wrong results. Generated corpora and typed outputs remain untracked under the version run tree.
* userspace/cuda: drop references to the removed local coordination noteArthur Hanson Rasmusson2026-09-23
| | | | | The per-area coordination file was local scratch (gitignored); future parallel work is announced through the project handoff instead.
* kernel: service BAR0/BAR1 region reads and extended config-space writesArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | Two more retry loops surfaced once the guest driver got past the CE binding: - BAR0 region reads returned -EINVAL ("would be handled by mmap"), so a QEMU-trapped read of BAR0+0x88084 failed and the driver polled it forever. Serve reads from the BAR0 shadow/trap handler (and mirror BAR1 reads from the emulated vidmem backing). - Config-space writes above offset 0xFF were refused with -EINVAL, so the driver's write to extended config 0x704 (PCIe device control area) never took effect and it retried. Accept and store extended-space writes; keep the legacy identity registers fixed.
* docs: refresh status, supersede stale material, link the CUDA analysisArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | | | | | | * Status now matches the live state: guest driver probe and nvidia-smi work; cuInit/device queries work; cuCtxCreate returns 999 because the emulated engine list is missing the async copy engine (COPY14). The userspace libcuda segfault is recorded as solved, with its mechanism (an RM allocation reply must echo request parameters; zeroing them corrupted the caller's saved frame pointer). * Correct the emulation claims: only a subset of RM controls are special-cased; the rest return a success reply with zeroed parameters whose consumed fields are still being corrected; allocation replies are coherent but no server-side object model exists. * Add the eight-point emulation checklist and reply-fidelity ladder as explicit gates, documented in userspace/cuda and linked from the architecture, implementation and operations docs. * Fix operational errors: build/test commands, module load order, mdev type names, LFS references, debugfs claims, in-repo firmware claims, CI claims, portability matrix (guest status, config layout, userspace driver facts). * Mark historical material: one superseded banner per legacy document with a pointer to its replacement; STATE.md marked as a 2026-09-21 snapshot. * Wiki: fix the index, add status banners and in-place factual corrections (msgq transport vs legacy mailbox, BAR definitions, error statuses, migration, glossary), and note the archived 2023 imports.
* userspace/cuda: add CUDA userspace driver analysis, tooling and name mapsArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | | Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs: * docs/01-08: architecture (layers, object model, threads, sync, memory, channels, launch), build and execution model, the driver/kernel/GPU interface contract, virtualization approaches, the 570.144 decompilation map with the corrected libcuda crash path and the current engine-topology blocker, tooling, and a version-upgrade playbook. * tools/: version-agnostic toolchain - stdlib ELF/interface toolkit (exports, unwind-table function map, wrapper resolution, version delta, crash address decode), callgraph walker, corpus annotator, and Ghidra analysis/decompilation/name-map scripts with resumable parallel drivers. * 570.144/annotations/: our name map (exports + resolved wrappers + verified internals) and subsystem index, tracked so naming work accumulates. * READ-ME-FIRST.md: reading guide with the fast lookup recipes. The full decompiled corpus and annotated corpus stay untracked (vendor- derived, gitignored under the version run/ tree).
* kernel: apply BAR0/BAR1 region writes to the emulated shadowsArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | The mdev's region write handler returned success without touching the BAR0 shadow or the trap table, so register writes that arrive through the VFIO region path (QEMU-trapped accesses) never took effect. A fresh guest then polled a register that never changed and spun forever in cuInit (QEMU log: vfio_region_write(mdev:region0+0x88704) failed: Resource temporarily unavailable). Route those writes to the BAR0 shadow/trap handler and mirror BAR1 writes into the emulated vidmem backing.
* kernel: coherent GSP-RM replies for engine-context creationArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | - GSP_RM_ALLOC: echo the request payload back with status forced to NV_OK. The kernel driver copies the reply params over the caller's params buffer using the class-expected size, so a zeroed reply smashed libcuda's stack (SIGSEGV at the engine-registration continuation); echoing makes the copy-back idempotent. - CE_GET_PHYSICAL_CAPS / CE_GET_ALL(_PHYSICAL)_CAPS: report sysmem, P2P and BL copy caps and mark the GR-synchronous CE (GRCE, COPY0). Without a GRCE the client's context walk ends with an empty secondary-engine record and the engine RM alloc is issued with class 0 (rejected -> cuCtxCreate fails). - TU104 engine table: CE0 shares the GR PBDMA ids (the driver's GR/GRCE runqueue validation requires it), plus an entry for the async copy engine (COPY14) advertised in the GSP static info engine caps. - tools/guest-diag/ioctl_trace: decode the RM status fields of alloc, control and free so the first failing driver call is visible in traces. gitignore: keep the CUDA analysis scratch tree (incl. WORKING-HERE.md) local.
* docs: public analysis describes components and behaviour onlyArthur Hanson Rasmusson2026-09-23
| | | | | | Move supporting detail to the adjacent local analysis notes (gitignored; referenced by name from the public analysis) and ignore those notes tree-wide. No behaviour or code changes.
* userspace/cuda: host golden-reference descriptor dump for the crash pathArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | The host libcuda (570.211.01) has the identical engine-registration helper instruction sequence (byte-pattern matched, host vaddr 0x454470 vs guest 0x453ef0). scripts/host_gdb_descriptor.sh runs cdt on the host under gdb and prints the per-engine 6-dword registration record and the dispatch hook target. Host registers a copy engine {3, 0xd, 4, 0xc5b5, 0x3e6, 3} and cuCtxCreate succeeds; the guest registers GR {1, 1, 0, 0xc5c0, 0, 0} and crashes - so the divergence is in the engine records the driver builds from the emulated GSP static info engine table.
* userspace/cuda: engine-list path summarised in the public analysisArthur Hanson Rasmusson2026-09-23
| | | | | | | | Records that libcuda builds an async copy-engine list from per-engine records (engine kind, runlist, class) received with the device information, which is derived from the GSP static info engine table this project emulates, and that the crashing per-engine helper operates on those records. The adjacent local analysis notes (userspace/cuda/570.144/LOCAL-ANALYSIS.md) carry supporting detail.
* userspace/cuda: analysis note for the cuCtxCreate segfault (gdb trace)Arthur Hanson Rasmusson2026-09-23
| | | | | | | | | Documents the exact crash: cuInit/device enumeration now succeed; cuCtxCreate faults in an engine-check/report path at libcuda+0x453f90 with rbp=0. Register trace shows rbp valid at the helper and at the dispatch hook's leave, but 0 immediately after the hook returns; the hook and the dispatch table entries are all valid libcuda code. Adds the ioctl sequence observed before the crash and lists what is still unanalysed.
* userspace/cuda: libcuda analysis tree (debug-focused), guest ioctl tracerArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | Analysis tooling for debugging CUDA execution in Open-IOV guests: - userspace/cuda/570.144/: fetch a library from a running guest (nc), objdump helpers, crash ip -> file vaddr + symbol mapping, and Ghidra headless import/analysis/decompile scripts (project kept so single functions decompile in seconds). The purpose is debugging correctness and performance of CUDA in Open-IOV VMs (not a reimplementation); extracted binaries and decompiled output are gitignored. - tools/guest-diag/ioctl_trace.c: ptrace tracker for the NVIDIA driver ioctls of a child (libcuda issues raw syscalls, so LD_PRELOAD sees nothing). Decodes RM_ALLOC (class) and RM_CONTROL (cmd) arguments; this is how the cuCtxCreate crash sequence was captured. - tools/test/cuda_drv_test.c: unbuffered + per-call markers so a crash identifies the CUDA call it happened in.
* kernel: execute pushed CE work (PTE memsets, copies) and track semaphoresArthur Hanson Rasmusson2026-09-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The progress service now decodes push buffers as method streams and performs the work the driver pushes, not just the tracking semaphore: - C5B5 method decoder: SET_REMAP_CONST_A/B + COMPONENTS + OFFSET_OUT + LINE_LENGTH_IN + LAUNCH_DMA = memset (into vidmem or guest RAM); OFFSET_IN + OFFSET_OUT + LAUNCH_DMA = copy; SET_SEMAPHORE_A/B/PAYLOAD + LAUNCH_DMA = release. Contiguous method runs (count 2/4) are applied in full - OFFSET_IN/OUT arrive as one run. - Pushes span page boundaries, so pages are decoded as method streams without a start marker. Collected operations execute only when the push's release is reached and still pending (payload ahead of the semaphore), so recycled buffers replaying old pushes are skipped and their stale memsets cannot overwrite newer state. - VA->PA map is first-wins: stale copies of freed allocations and unrelated adjacent u64 pairs produced bogus entries that hijacked a good mapping and sent semaphore writes to the wrong page. - dump_guest module parameter (hex dump of guest PA, kthread via completion; kthread_stop before the thread runs loses the work). With this the guest no longer stalls in uvm_tracker_wait and no longer oopses in the UVM path; the failure moved to a userspace segfault inside libcuda (NULL-0x38) after the driver's page allocations. Also add tools/cuda-ab: run one deterministic CUDA workload on the host (bare metal oracle) and in the guest, then compare the stable result lines. The binary links the CUDA runtime statically so the guest only needs the driver's libcuda.so.
* kernel: semaphore service forward-jump completion, stats, sema_probeArthur Hanson Rasmusson2026-09-23
| | | | | | | | | Relax the in-order rule to forward progress (payload > current), with the jump bounded and counted: UVM tolerates forward jumps up to 2M (2 * UVM_CHANNEL_NUM_GPFIFO_ENTRIES_MAX), and push buffers are recycled so intermediate pushes can be missed. Add miss/stale/jump counters to the rebuild log and a sema_probe module parameter to dump the service's map entry and live slot values for a GPU VA (diagnostics for tracker stalls).
* Untrack legacy trees and purely legacy docs (kept locally)Arthur Hanson Rasmusson2026-09-23
| | | | | | | | | - patches/ -> legacy-patches/ (gitignored) - legacy-driver-guest/ (old guest vGPU driver, gitignored) - docs/old/ (superseded architecture write-up, gitignored) - docs that only reference the legacy components stay tracked; their path references now point at legacy-patches/ - install.sh PATCH_DIR updated
* Open-IOV: initial public releaseOpen-IOV2026-09-23
Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-13 02:42:26 +0000