summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* guest-diag: add --fwsec to the supervised claim harness (stage-3c write run)Arthur Hanson Rasmusson11 days
|
* gsp-phys: correct the FWSEC interface-patch size and staging loggingArthur Hanson Rasmusson11 days
|
* gsp-phys: read-only FWSEC dry-run mode; match the 6-byte BIT token decodeArthur Hanson Rasmusson11 days
|
* STATE-CURRENT: supervised D0 claim mitigation verified (3/3); FB 8 GiB; ↵Arthur Hanson Rasmusson11 days
| | | | FWSEC dry-run stage
* STATE-CURRENT: 07:12 check-back snapshot (backend hardening/read-only ↵Arthur Hanson Rasmusson11 days
| | | | dry-run stage before live boot)
* gsp-phys: fix the WPR2-up release hand-back and tighten FWSEC/VBIOS checksArthur Hanson Rasmusson11 days
|
* STATE-CURRENT: PLAN B claim path proven safe 3/3; real FB size 8 GiB; ↵Arthur Hanson Rasmusson11 days
| | | | secure-verification stage coded (build-only)
* gsp-phys: stage 3c FWSEC via VBIOS BIT image plus failure recoveryArthur Hanson Rasmusson11 days
|
* gsp-phys: decode the real FB size for the boot layout (stage 3b-prime)Arthur Hanson Rasmusson11 days
|
* STATE-CURRENT: claim harness verified; rung-3 boot spec complete; FB-size ↵Arthur Hanson Rasmusson11 days
| | | | decode identified
* guest-diag: reusable supervised physical-claim cycle harness (D0 pre-lock + ↵Arthur Hanson Rasmusson11 days
| | | | abort checks)
* gsp-phys: hold a GPU runtime-PM reference across the claim window (udev ↵Arthur Hanson Rasmusson11 days
| | | | bind-event hardening)
* STATE-CURRENT: D0 lock verified 3/3 supervised claim/release cycles; real ↵Arthur Hanson Rasmusson11 days
| | | | GSP boot is next
* gsp-phys: harden the D0 lock (pre-lock readiness, no unlock without rebind, ↵Arthur Hanson Rasmusson11 days
| | | | strict board check)
* STATE-CURRENT: power-domain lock implemented behind the hazard gate; ↵Arthur Hanson Rasmusson11 days
| | | | snapshot refreshed
* docs: refresh status wording after the 64/64 baseline and the ↵Arthur Hanson Rasmusson11 days
| | | | physical-backend lock
* gsp-phys: keep the dGPU power domain in D0 across the claim/release handoffArthur Hanson Rasmusson11 days
|
* docs: record the board power-domain hazard and D0-lock mitigation for the ↵Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | physical backend - New 4.2: describe behaviourally how the option-D claim path can power-gate the board on driver release (platform power resource behind the root port, off/on sequence via the embedded controller), the two observed host freezes (no crash record), and the D0 lock now applied across release -> claim -> rebind: runtime-PM forbid + D3cold disable on the GPU, its board companion functions and the root port, config-space verification while claimed and again before rebind, abort with the locks held on any power-loss signature, runtime-PM restore after the distribution driver's probe. - 4.1/4.3: the lock is a required ownership step; returning to the normal driver is conditional, not assumed; real GSP boot not started. - 2/3(A): fresh no-unbind evidence - no exported RM/vGPU symbols and no manager daemon in the stock distribution build. - 6: driver-API baseline re-verified 64/64; ownership + firmware staging verified; power-domain lock implemented pending supervised verification. - 7/8: residual out-of-band power-cut risk, supervised-test requirement, and the gsp_phys_gpu + gsp_phys_hazard_ack operating constraint.
* STATE-CURRENT: cdt2 baseline re-verified; second freeze class mechanism and ↵Arthur Hanson Rasmusson11 days
| | | | power-domain mitigation recorded
* STATE-CURRENT: execution backend hazard-gated after freeze root-cause; ↵Arthur Hanson Rasmusson11 days
| | | | bring-up snapshot refreshed
* STATE-CURRENT: root-cause the second freeze class (claim unbind) and its ↵Arthur Hanson Rasmusson11 days
| | | | hazard gate
* gsp-phys: stage-3b boot layout plus a hazard gate for the claim pathArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | Stage 3b allocates the three coherent boot-argument pages and fills the 256-byte WPR metadata from the computed Turing FB layout; no device registers are written (uncommitted work from the previous session, committed as-is behind the new gate). The physical claim unbinds the distribution NVIDIA driver; on this host that power-gates the dGPU board and the re-bind can leave the GPU unresponsive and hard-freeze the system (observed twice, 2026-09-30, no crash record). The claim now refuses unless the new gsp_phys_hazard_ack module parameter is also set, so a stale command line cannot re-trigger it. See the hazard note at the top of gsp_phys_gpu.c.
* gsp-phys: boot-layout stage 3b - boot-argument pages, WPR meta and FB layout ↵Arthur Hanson Rasmusson11 days
| | | | (no device writes)
* STATE-CURRENT: execution-backend bring-up started (claim + GSP firmware ↵Arthur Hanson Rasmusson11 days
| | | | verified); freeze-class watch note
* gsp-phys: read-only falcon state probe for the boot rungArthur Hanson Rasmusson11 days
| | | | | | | Logs both falcons' run/halt bits, the RISC-V active bit, WPR2 up/down and the security-falcon mailboxes when the physical GPU is claimed. Verified live: GSP and SEC2 halted, RISC-V inactive, WPR2 down (HI=0). This is the read-only half of the real boot rung; no device state is written.
* docs: execution-backend bring-up staged - ownership and firmware staging ↵Arthur Hanson Rasmusson11 days
| | | | verified, real boot next
* STATE-CURRENT: cdt2 64/64 rerun verified; execution backend bring-up ladder ↵Arthur Hanson Rasmusson11 days
| | | | drafted
* gsp-phys: claim the physical GPU and stage real GSP firmware (backend rungs 1-2)Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | Rung 1 (ownership): a default-off gsp_phys_gpu module param; on claim the distribution driver is released from 10de:1ed3, our pci_driver binds it, BAR0/1/3 are mapped and the falcon state is sampled; release unbinds ours and rebinds the distribution driver. Verified live: claim reads BOOT0=0x164000a1 with the falcon halted and mailboxes clear; rmmod rebinds nvidia and nvidia-smi stays healthy. Rung 2 (firmware): while the device is owned, gsp_phys_fw (default nvidia/570.211.01/gsp_tu10x.bin) is fetched with request_firmware, checked as ELF64 RISC-V, and its sections are logged (.fwimage, .fwversion, .fwsignature_tu10x). The image is held for the boot rung. The claim is default-off; the emulation path is unchanged.
* sema-svc: rewind replays may only be skipped when their release was deliveredArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | The per-put-epoch consumed-slot skip must not suppress a slot whose entry was consumed *without* delivering its release (the CE grace expiry in real mode): the driver-API smoke test's CE pushes are consumed unreadable before their ring pin lands and depend on the pin-batch rewind replaying them once the translation appears. The skip now applies only to slots whose consumption delivered a release: - tail consumption records a slot when the iteration decoded a release (svc_rel_count delta); the sweep-delivered entry_want consume and the fake-release consume always record - the grace-expiry consume records only when a recovery (fake or sema_rel_recover) delivered the release - svc_fake_release counts as a delivered release So real-mode replays still re-process undelivered entries, while the runtime-client case (readable pushes delivered immediately) keeps the fast path that removed the 8 s stalls.
* STATE-CURRENT: real-release rc=0 chain reached; execution backend is the ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | frontier - the per-push 8 s stall was pump bookkeeping (release-seen keyed on the unchanged tracker-semaphore VA; rewind replays returning to the CE grace path) - fixed; real-release init now completes in ~1 min - the content rescue now parses the real method encoding, so NON_INC-headed pushes validate and their releases land - the queued launch's module image reaches the emulated vidmem; execution on real hardware (docs/03-architecture/08 backend) is the next project
* sema-svc: detect a decoded release by count, not by the (unchanging) VAArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | The pump decided "this push carried a release" by comparing the last decoded release VA before/after svc_decode_stream. Consecutive pushes on a channel release to the SAME tracking-semaphore VA (UVM's tracker pattern), so every push after the first read as release-less: the entry was held for the full CE grace window (observed: exactly one push consumed per 8 s, gating the whole runtime init; the release itself had already been applied by the decode, only the pump's bookkeeping stalled). svc_rel_count now increments on every decoded release (host sem block, REPORT_SEM block, CE launch-DMA release) and the pump compares counts at all three sites (initial decode, rescue re-route, rescued-page retry).
* sema-svc: never re-process a GPFIFO slot already consumed in the put epochArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | A pin-batch rewind replays recently consumed entries. An old entry whose rescue cannot validate any candidate then sits in the CE grace path - and because the client's put has advanced since, the replayed entry no longer matches any "new" expectation: it waits the full 8 s grace, blocking every later entry behind it on that channel. Observed live: the first push of rank 0 re-processed and skipped twice, 38 s apart, while eight later pushes queued behind it. The pump now marks each consumed slot in a per-channel bitmap valid for the current put epoch (cleared when put wraps, so a recycled slot in the next epoch is processed normally) and consumes a replayed slot immediately. Rewinds still refresh readable entries; they simply no longer resurrect entries that were already handled. Evidence: real-release watcher cycle, replay-after-put-advance no longer produces 'skipping stale entry ... after 8000 ms grace' for old slots.
* sema-svc: consume rewind replays and sweep-delivered CE entries at onceArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | | Two stalls serialized the runtime client's init into one 8 s CE grace window per push: - a pin-batch rewind replays recently consumed GPFIFO slots; re-examining a consumed slot runs the rescue and then holds it for the full grace window again (observed: the same VA 'skipped after 8000 ms grace' twice, 38 s apart). The pump now records each consumed slot with the put value it was consumed at (32-entry per-channel replay ring, persisted across passes) and consumes a replayed slot whose producing put has not advanced immediately. - the content sweep applies releases from RAM pages before the pump reads their entries, so the pool advances past the entry's payload while it is pending. entry_want (recorded on first examination) now detects that on both the unreadable-rescue and the readable-no-release paths and consumes the entry instead of waiting the grace window. Evidence: real-release watcher cycle, pool semaphore advances no longer gated at 8 s intervals once the replay skip engages.
* sema-svc: validate pushes with the full method encoding; deliver releases ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the sweep already applied The runtime client's second push on each UVM copy channel starts with a NON_INC method (0x60080002, count 8) and its release method sits 0x50 bytes in. The stream validator accepted only INC (op 1) headers, so the readable page failed the plausibility gate and every content-rescue candidate for that VA failed the read-back validation - the client's tracking semaphore for the push never advanced and the runtime init parked. - one shared parser (svc_stream_parse) accepts INC/NON_INC (op 1/3), IMMD_DATA (op 4, header-only), ONE_INC (op 5) and END_PB_SEGMENT (op 7), and still requires an exact method boundary over the whole entry plus trailing zero padding - the strict end condition is what keeps the content rescue safe against firmware/allocator pages - svc_method_stream_plausible and svc_stream_full_ok both use it, so a real readable push is decoded directly and a rescued page validates - the decoder skips NON_INC/ONE_INC methods whole (including their data words) instead of resyncing word-by-word through the payload, which could misread data as headers and queue bogus work Follow-on: the content sweep applies releases from RAM pages ahead of the pump, so the pump's expected payload was already satisfied when it examined the entry; the CE grace path then held each entry for the full 8 s window before consuming it - one grace period per push, serializing the client's runtime init. The pump now records the expected payload when it first examines an entry (entry_want) and consumes the entry immediately once the pool advances past it, on both the unreadable-rescue and the readable-no-release paths. Evidence (fresh boot, real releases, watchers + kprobes, no fakes): cudaGetDeviceCount rc=0 in the runtime client, device-attribute walk proceeds, pool semaphores advance; before this change the client sat in the 0x20801702 service-interrupt loop for >6 min and never left device init.
* STATE-CURRENT: no-pin release delivery committed; real-release init reaching ↵Arthur Hanson Rasmusson12 days
| | | | | | | | device count - 7f84797 delivers UVM tracking releases without host-side pins - one real-release no-watcher cycle reached cudaGetDeviceCount success - cycles continue; execution backend remains the gating item
* sema-svc: deliver UVM CE tracking releases without host-side pinsArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The runtime client's registration stalled when no guest-side pin scanner was running: the content rescue refused to run before the pool semaphore was translated (its expected payload was unlearned), and unreadable CE entries were consumed after two failed rescues ~40 ms apart, dropping the client's wait. The push pages then could not be found within the entry's grace window. - rescue: drop the learned-payload gate (the matcher only needs the channel rank and the release shape); keep unreadable CE entries pending for sema_ce_grace_ms instead of consuming after a fixed attempt count - expected payload: re-learn from the pool whenever it advanced (a deferred release could land after the first read, leaving the expectation stale) - matcher: require the expected payload exactly and validate the read-back stream carries it, so a recycled page of an older push cannot be pinned - scan: descending full-range pass (the work allocations live in the high RAM block) plus a +/-512 MB near pass around the verified push-buffer base - releases decoded with an untranslated semaphore are queued and applied when the page appears (svc_pending_add on both decode paths) - a CE push that reads as a plausible stream but carries no release for the channel is re-routed through the rescue instead of silently consumed - new module param sema_rel_recover (default off): after grace + rescue, apply the channel's expected tracking value so the client's wait clears; counter-visible as releases_recovered, distinct from sema_fake_rel Evidence: fresh boot, real releases, no host watcher, no kprobes - the six UVM CE first pushes are found by content rescue, their releases applied, and the client reaches cudaGetDeviceCount rc=0 (it then waits for the third push, which this change routes to the rescue).
* STATE-CURRENT: no-watcher cycles iterating (module rebuilt from the ↵Arthur Hanson Rasmusson12 days
| | | | | | | in-flight edit) - three real-release cycles without kprobes/watchers; no textual result yet - frontier unchanged: real-release runtime init and the execution backend
* STATE-CURRENT: activity resumed - no-watcher real-release runtime cyclesArthur Hanson Rasmusson12 days
| | | | | | - fresh module per cycle without kprobes/watchers; early observation: a compute-channel entry stalls with a failure notifier - frontier unchanged: real-release runtime init and the execution backend
* STATE-CURRENT: runtime client reaches kernel launch; execution backend is ↵Arthur Hanson Rasmusson12 days
| | | | | | | | the next wall - runtime init rc=0 (cudaGetDeviceCount/SetDevice/alloc/HtoD/launch accepted) - fixed the legacy clock-domain probe replies that blocked runtime init - next: real-release runtime init and physical-GPU execution of queued work
* GSPRM CLK legacy controls: populate the domain queries the runtime clock ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | | | helper gates on The static CUDA runtime's device init failed with cudaErrorInitializationError even though every kernel ioctl returned 0: cuInit succeeded and then cuDeviceGetAttribute(ECC_ENABLED=36) returned CUDA_ERROR_NOT_FOUND (500). Tracing libcuda's clock-domain helper (libcuda+0xe8c410) shows it issues the CLK legacy controls 0x20809009/0x20809064/0x20809001 and bails with 500 when the replies do not carry the expected domain bits/entries; the dependent attribute getters (ECC_ENABLED, TOTAL_CONSTANT_MEMORY) then abort init. Reply to the three queries with the minimal fields the helper checks: bits 32|34 present in 0x20809009, non-zero domain entries at offsets 12/20/28/36 of 0x20809064, and a matching selector in 0x20809001. No frequencies are fabricated (all other fields stay zero). Verified in the guest with the stock 570.144 static-cudart vectorAdd client: cudaGetDeviceCount -> rc=0 (ndev=1), cudaSetDevice, cudaFree, three cudaMallocs and both HtoD copies all return rc=0; the client then parks in cudaDeviceSynchronize waiting for the (not yet forwarded) kernel launch.
* STATE-CURRENT: PD writes proven live; runtime client past registration; ↵Arthur Hanson Rasmusson12 days
| | | | | | | | canonical reply fix - page-directory entry writes captured and validated; 64/64 baseline stays - vectorAdd client now fails after registration in libcuda (rc=3), no kernel error - BAR-info canonical reply-size bug fixed; one cache mismatch remains
* guest-diag: PD-watch cycle (dst-keyed root-PD watch + park-time shadow dumps)Arthur Hanson Rasmusson12 days
| | | | | | | | Reproducible TASK 2 harness: fresh boot, sema_dbg=Y with sema_copy_dst=0x200000 / sema_copy_dst2=0x208000, cdt2 parked at CDT_CP_PAUSE so the root PD/PT pages can be dumped and walk_probe run while the client is live. Prints the offending write data (prov= names the translation layer) and the post-run walk state.
* GSPRM BUS_GET_PCI_BAR_INFO: reply the canonical 8-entry params structArthur Hanson Rasmusson12 days
| | | | | | | | | | The control is cacheable in the guest kernel (RMCTRL_FLAGS_CACHEABLE) and rmapiControlCacheSet validates paramsSize against the canonical sizeof(NV2080_CTRL_BUS_GET_PCI_BAR_INFO_PARAMS) = 200. We replied only 8 + 4*24 = 104 bytes, so every reply tripped NV_ERR_INVALID_PARAMETER in rmapiControlCacheSet (rpc.c:9998) and the BAR-info cache entry was never populated. Zero-fill and report all eight NV2080_CTRL_BUS_PCI_BAR_INFO entries (count still 4).
* STATE-CURRENT: quiet logging landed; baseline verified twice; PD-capture ↵Arthur Hanson Rasmusson12 days
| | | | | | | | task next - dmesg diagnostics gated behind default-off sema_dbg; 64/64 with a 520-line log - staging buffer kept resolvable via streamed pins; PD-entry capture remains - note the guest-side kernel fault observed in serial
* guest-diag: stream the client's high-VA UVM sysmem pages as pinsArthur Hanson Rasmusson12 days
| | | | | | | | | The DtoH staging buffer lives in the client's high-VA /dev/zero|nvidia mappings (e.g. devptr+0x3c00100), outside the scanner's fixed CE/GR windows; only the host's pagemap read can see it. The guest scanner now walks those VMAs and streams their present pages so the module pins them. run_cycle.sh learns the sema_dbg master switch (auto-on when any watch knob is set).
* sema-svc: gate per-push/per-doorbell/RPC tracing behind default-off sema_dbgArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | A healthy run left dmesg dominated by per-operation lines (sema-push, sema-lda, sema-chan rel-apply, sema-grce, DOORBELL/BAR writes, GSPRM per-RPC lines, the teardown garbage-element stream). Introduce the module parameter sema_dbg (default 0) and route every high-frequency diagnostic through it; the finer-grained watch knobs (sema_ops_log, sema_copy_watch, sema_copy_pat, sema_copy_dst, sema_copy_dst2) only take effect while it is on. Boot stats, channel registrations, pin events and state changes stay unconditional. Also carries the per-side translation provenance (op_tr_src, printed as prov= in the destination-window watch) and the 16 KB destination window used to observe the guest's page-table writes. Verified: fresh boot + full cdt2 smoke test (64/64, ALL OK) leaves a 520-line dmesg with zero per-push/per-doorbell/per-RPC lines.
* STATE-CURRENT: payload roundtrip 64/64 restored by the page-table ↵Arthur Hanson Rasmusson12 days
| | | | | | | | publication capture - the device-to-host destination now resolves; baseline green again in live runs - frontier/active-work notes updated: land the capture fix, then the runtime vectorAdd client
* STATE-CURRENT: baseline regression (DtoH destination translation) and the ↵Arthur Hanson Rasmusson12 days
| | | | targeted-fallback/retry work
* sema-svc: hold and retry a readable push whose operand is under a targeted ↵Arthur Hanson Rasmusson12 days
| | | | RAM scan (the DtoH dst staging VA needs multiple scan slices)
* sema-svc: operand resolver gains the targeted rm_mem fallback; per-side ↵Arthur Hanson Rasmusson12 days
| | | | translation-miss tracing (the DtoH copy operand is the failing side)
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-12 15:09:09 +0000