From c9bf1cf098c8ea4daeb3e6c9b274a9025c03045d Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 15:13:06 -0400 Subject: kernel: GR context-promote bisection modes 4-17 and server-managed-ctxbuf registry variant; complete vgsp rename; docs/tooling refresh - gsp_phys_booter: GR promote bisection modes 4-17 (two-phase zero-field VA, batched entries, hVirtMemory, bare VA, KMD SKIP_CTXBUFFER_ALLOC channel, old-interface GPU_INITIALIZE_CTX, FBMEM context buffer, UVM-faithful PA-only, PATCH/PM auxiliary promotes, server-managed no-promote path); PATCH/PM geometry queries and buffers; pre-boot SET_REGISTRY profile selection - gsp_phys_gpu: gsp_phys_rmcall_gr_promote clamp 0..17 and gsp_phys_srv_ctxbuf parameter; registry image variant with RMSetClientRMAllocatedCtxBuffer=0 - vgsp rename across kernel, scripts, tools and docs (module vgsp.ko, /sys/kernel/debug/vgsp/, vgsp-: log prefixes) - risc-v-gsp-rm-debugger user-facing rename and docs/CI refresh; guest-diag tool updates; README/SBOM/install/Makefile sync Live status: CE ladder and P6 guest-copy forwarding green; GR engine context commit still open (all promote shapes accepted or cleanly refused; the channel stays CTXNOTVALID) - see Handoff-vGPU-IO.md Addendum 21 (local). --- docs/eng-ops.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) (limited to 'docs/eng-ops.md') diff --git a/docs/eng-ops.md b/docs/eng-ops.md index 218b8cd..20985d7 100644 --- a/docs/eng-ops.md +++ b/docs/eng-ops.md @@ -198,7 +198,7 @@ flowchart LR HR["harness/*.py"] end subgraph HOST["Host and guest"] - MOD["open-iov-vfio.ko"] + MOD["vgsp.ko"] VM["stock guest + guest-diag"] RMH["rmtool/rmtop/rmq + rmharness"] end @@ -450,7 +450,7 @@ decision. **Caveat, in every package README: this is a simulation package, not a signed production image.** It is not a firmware patch, must never be flashed or injected into a guest, and is not bootable on silicon. Injected code is development-only ([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) ยง1); -productization goes through `open-iov-vfio` + physical confirmation, never +productization goes through `vgsp` + physical confirmation, never through a simulator package. Local-only, budgeted, session-recorded; never touches `/dev`, a module, an @@ -550,7 +550,7 @@ the rules win. **Build order (physical-backend work needs the kernel module only):** -1. Build `open-iov-vfio.ko` first; it generates the `Module.symvers` the +1. Build `vgsp.ko` first; it generates the `Module.symvers` the optional OpenRM modules link against. `scripts/stack_manager.sh build-kernel`. 2. Build the optional OpenRM `nvidia.ko` with @@ -566,7 +566,7 @@ the rules win. 1. Prerequisites: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `mdev` (`scripts/stack_manager.sh load-prereqs`). -2. `open-iov-vfio.ko` with the exact reviewed parameter set +2. `vgsp.ko` with the exact reviewed parameter set (`scripts/stack_manager.sh load-open-iov`). The backend claim is **module-owned** at load with `gsp_phys_gpu=1 gsp_phys_hazard_ack=1`, or **mdev-owned** and deferred to the first mdev create with @@ -602,7 +602,7 @@ authoritative for the literal default. | `gsp_phys_fault_relay` | 0 | On a relay timeout, drain the status queue for the engine fault (`0x1006`, corroborated by `0x1020`), match runlistId/chid and write the guest channel notifier + event. | Requires `gsp_phys_relay=1` and the three parameters above. | | `gsp_phys_fault_inject` | 0 | Debug: write the relayed pair's source PTE as zero to force an engine fault on the next relay submission; only relayed pairs are affected. | Requires the relay path; default 0 programs the real source HPA. | | `gsp_phys_rmcall_engine` | 0 | Ladder engine: 0 = COPY0 (CE copy submission), 1 = GR0 (compute object, BIND engineType 1, GR semaphore release). Snapshotted when stage 3 starts; mid-session changes are ignored. | Requires boot + init-done; CE forwarding paths refuse on a GR ladder. | -| `gsp_phys_rmcall_gr_promote` | 0 (clamp 0..3) | GR context-promote bisection: **0 never sends `PROMOTE_CTX`** (the ladder stops after the geometry queries, context buffer/PT allocation, PDB mapping and channel/compute alloc, before BIND); 1 = single MAIN entry, initialize/no-nonmapped, size raw+subctx; 2 = two-phase PA-only then VA-only; 3 = single entry with the exact raw size. Nonzero enables can assert GSP-RM (`0x52a8b04`) and wedge the device. | GR ladder only; value snapshotted at stage 3; **requires confirmed reboot readiness** before enabling. | +| `gsp_phys_rmcall_gr_promote` | 0 (clamp 0..17) | GR context-promote bisection: **0 never sends `PROMOTE_CTX`** (the ladder stops after the geometry queries, context buffer/PT allocation, PDB mapping and channel/compute alloc, before BIND); 1 = single MAIN entry, initialize/no-nonmapped, size raw+subctx; 2 = two-phase PA-only then VA-only with the same PA/VA/size; 3 = single entry with the exact raw size; 4 = two-phase with the reference client's zero-field VA-only step; 5 = one control with `entryCount=2` (PA-only init + zero-field VA-only); 6 = no-entry `hVirtMemory`: allocate an `NV01_MEMORY_VIRTUAL` (class 0x70) VA-range object on the ladder VAS at the context window, then `PROMOTE_CTX` with `hVirtMemory`/`virtAddress`/`size` and `entryCount=0`; 7 = no-entry bare `(virtAddress, size)` with `entryCount=0` and no `hVirtMemory`; 8 = single MAIN entry like 1 on the KMD-managed channel variant V6 (the V5 channel plus alloc flag `SKIP_CTXBUFFER_ALLOC` bit 31); 9 = same V6 channel with the stock client's zero-field VA-only MAIN entry (`bInitialize=0`); 10 = same V6 channel, `GPU_INITIALIZE_CTX` (`0x2080012d`) installs the physical context memdesc first, then `PROMOTE_CTX` `entryCount=0` bare `(virtAddress, size)`; 11 = mode 10 with the context buffer swapped to a server FBMEM allocation (class 0x40, VIDMEM `physAttr=0x4`, aperture-0 leaf PTEs); 12 = V5 UVM/vGPU-faithful PA-only MAIN entry (`bInitialize=1 bNonmapped=1`, no VA) and continue to BIND/SCHEDULE/submit; 13 = mode 12 plus a PATCH PA-only promote first (size from engine context properties engineId 0x10); 14 = V6 KMD channel + PATCH PA-only then the single-entry MAIN init+VA (the flow that books the context VA and calls the commit); 15 = mode 14 with a server FBMEM MAIN buffer and VIDMEM `physAttr 0x4`; 16 = mode 14 plus a PM ctxsw PA-only promote first (engineId 0x09); 17 = no client promote, paired with `gsp_phys_srv_ctxbuf=1` (pre-boot registry profile with `RMSetClientRMAllocatedCtxBuffer=0`) so the GSP-RM server-manages the context buffers. Live result: modes 1, 2 step 2 and 4 all wedged GSP-RM (`0x52a8b04`, `NV_ERR_OBJECT_NOT_FOUND` 0x57 in the LOGINIT stack) while the PA-only nonmapped step of modes 2/4 is accepted every time - the VA resolve, not the entry fields, is the wall (our external VAS has no server-side VA booking). Mode 6 (hVirtMemory, V5 channel) was refused **cleanly** (`status=0x1f` `NV_ERR_INVALID_ARGUMENT`, no assert, clean release) because the channel was not KMD-managed. Modes 8-11 all completed with **clean releases** (WPR2 down, driver rebound, 5 locks, pstore empty): the V6 channel alloc is accepted (`flags=0x80800000`); modes 8 and 9 (with and without an init entry) both return a clean `status=0x40` `NV_ERR_INVALID_STATE` (mode 9 without the server error log - entry shape irrelevant); mode 10's `GPU_INITIALIZE_CTX` is accepted (`status=0`) but the bare-VA promote still returns `0x40` with the same server log anchor; mode 11 (server FBMEM context buffer) returns `0x40` with a **different** server log anchor, so FBMEM moves the failure to a different structural check. Session 2 (modes 12-17, all clean): mode 12's V5 PA-only MAIN promote is **accepted** (`status=0`) and BIND succeeds, but the GR push faults `CTXNOTVALID` (`sem=0`, `GPGet=1`) - the accepted entry does not set `ENG_CTX_VALID`; mode 13 (PATCH first) and mode 14/15/16 (V6 KMD + PATCH, +PM, FBMEM/VIDMEM) all leave the MAIN promote at the same `0x40` rec-677 site as mode 8, so PATCH/PM/buffer type are not the missing prerequisite; mode 17 (registry `RMSetClientRMAllocatedCtxBuffer=0`, no promote) also faults `CTXNOTVALID`, so even server-managed context buffers leave the channel invalid - the remaining wall is the engine-context commit/RAMFC path, not buffer ownership. Nonzero enables can still assert GSP-RM and wedge the device. | GR ladder only; value snapshotted at stage 3; **requires confirmed reboot readiness** before enabling. | | `gsp_phys_repower` | 0 (latched) | Restore the pre-claim runtime-PM/D3cold policy when releasing the GPU; with 0 the D0 locks and runtime reference are deliberately left held until reboot. | **Never enable** (rule 3); re-arming D3cold after a claim can hard-freeze the host. | The emulation-side `sema_*` parameters (ring discovery, release recovery, @@ -627,7 +627,7 @@ hard signature. **Step 1 โ€” build and identity check.** Rebuild the reviewed source; record the module hash; never run a cycle against a stale or unknown module. -**Step 2 โ€” load.** Prerequisites, then `open-iov-vfio.ko` with the reviewed +**Step 2 โ€” load.** Prerequisites, then `vgsp.ko` with the reviewed parameters. Module-owned: the claim and requested stages run at load. Mdev-owned: load, then create the mdev (the async claim/boot starts) only when no VM is live. @@ -699,7 +699,7 @@ written post-reboot plan - never a second attempt in the same boot. ### 10.4 debugfs inventory All files live under the module's debugfs directory -`/sys/kernel/debug/open-iov-vfio/`. **The directory and every file are +`/sys/kernel/debug/vgsp/`. **The directory and every file are root-only on this system**: existence and read checks must run as root (a permission error can look like an absent file). Every writer is inert unless its gates hold, and every accepted write performs a real RPC or device @@ -760,11 +760,11 @@ bump, device/ISA/ROM change, protected state, matrix/CI) are owned by | Symptom | Exact signal to look for | Decode / likely cause | First diagnostic | Action | |---|---|---|---|---| | Guest GR push faults | `func=0x1006 ... exceptType=0x20 runlistId=0 chid=1 errString="... intr1 80000000"`; NOCAT `recType=4 errorCode=0x20 faultingEngine="GR"` | `0x20` = robust-channel PBDMA error; `intr1` bit 31 = `CTXNOTVALID`; the external-VAS GR engine context was never committed (`ENG_CTX_VALID=0`), so the guest binds via `PROMOTE_CTX` (`0x2080012b`). CE channels are unaffected. | `gsp-phys-status-drain`; `gsp-phys-state` `rm_call`/`rm_call_ctx` | Decode first, no retry. Only a proven promote shape commits the context; keep the client promote path gated by `gsp_phys_rmcall_gr_promote=0` until a variant is proven. | -| `PROMOTE_CTX` wedge | `gsp-nocat: rec seq=N recType=5 errorCode=0x52a8b04 source="ASSERT" faultingEngine=""`, then no fn76 reply (timeout `-110`); status ring stops, server command read pointer advanced | A host/server-RM assert in the promote path - not an engine fault (no `0x1006`). The server consumed the command and wrote no reply. Teardown cannot clear WPR2 (its unlock is fuse-locked on this part). | `gsp-phys-state` (`cmd_tx`/`stat_tx`, `wpr2`), release log | Ladder aborts by design; **do not retry**. Fail-safe holds the device; reboot with written approval (rule 2); bisect exactly one promote variant per cycle after reboot. | +| `PROMOTE_CTX` wedge | `vgsp-nocat: rec seq=N recType=5 errorCode=0x52a8b04 source="ASSERT" faultingEngine=""`, then no fn76 reply (timeout `-110`); status ring stops, server command read pointer advanced | A host/server-RM assert in the promote path - not an engine fault (no `0x1006`). The server consumed the command and wrote no reply. Teardown cannot clear WPR2 (its unlock is fuse-locked on this part). | `gsp-phys-state` (`cmd_tx`/`stat_tx`, `wpr2`), release log | Ladder aborts by design; **do not retry**. Fail-safe holds the device; reboot with written approval (rule 2); bisect exactly one promote variant per cycle after reboot. | | WPR2 / clean-release failure | `teardown did not clear WPR2 (HI=0x...)`; `release: WPR2 still up after teardown ... refusing the hand-back`; `NOT rebinding ... power domain stays LOCKED` | The booter-unload gate bits were never completed by the wedge, or the board stopped answering; the host-side unlock is fuse-locked. | `gsp-phys-state` `wpr2`, `scratch14`, `stage:` | Fail-safe is the designed state: no rebind, locks held, WPR2 up. Leave the device; reboot with approval. | | Module unload hang | `rmmod` does not return; `ps` shows the emulation/relay kthreads or work items; mdev still present | Teardown ordering (kthread/workqueue) or an open mdev/VM; the emulation-side cleanup applies. | `ps -e \| grep gsp_emu`; mdev list; `dmesg` tail | Remove mdev/stop VM first; `scripts/stack_manager.sh cleanup-crash`; reboot with approval if it persists (rule 5). | -| Guest `cuInit` wedge / guest flake | guest `cuInit rc=999`, `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`, or a host `gsp-phys-fwsec: unexpected WPR2 already up`; a one-off guest Oops in `free_swap_cache` cleared on retry | On the emulated path this is a known guest-side flake; the WPR2-up line is the FWSEC refusal on a board already in a bad state. | guest dmesg; host `gsp-phys-state` `wpr2`; retry the guest once | If only the guest flaked, restart the guest once. If `unexpected WPR2 already up` appears, stop the cycle and inspect state; never retry FWSEC (rule 4/5). | -| GSP boot failure | `gsp-phys-fwsec: staging not visible before kick (DMEM0 ... IMEM ...)` then `FWSEC/FRTS execution failed: -5`; historical `GSP_INIT_DONE rpc_result=0x59` | Staging DMA did not land (async mdev-owned FWSEC path) or, historically, the pre-boot handshake/identity payload was wrong (`0x59` = operating-system error). | `gsp-phys-state` `stage:`, `gsp-phys-log-dump` | Use the module-owned claim path (proven); the async FWSEC defect is fail-open and off the critical path. Do not repeat a failed boot in the same boot after a hard signature. | +| Guest `cuInit` wedge / guest flake | guest `cuInit rc=999`, `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`, or a host `vgsp-fwsec: unexpected WPR2 already up`; a one-off guest Oops in `free_swap_cache` cleared on retry | On the emulated path this is a known guest-side flake; the WPR2-up line is the FWSEC refusal on a board already in a bad state. | guest dmesg; host `gsp-phys-state` `wpr2`; retry the guest once | If only the guest flaked, restart the guest once. If `unexpected WPR2 already up` appears, stop the cycle and inspect state; never retry FWSEC (rule 4/5). | +| GSP boot failure | `vgsp-fwsec: staging not visible before kick (DMEM0 ... IMEM ...)` then `FWSEC/FRTS execution failed: -5`; historical `GSP_INIT_DONE rpc_result=0x59` | Staging DMA did not land (async mdev-owned FWSEC path) or, historically, the pre-boot handshake/identity payload was wrong (`0x59` = operating-system error). | `gsp-phys-state` `stage:`, `gsp-phys-log-dump` | Use the module-owned claim path (proven); the async FWSEC defect is fail-open and off the critical path. Do not repeat a failed boot in the same boot after a hard signature. | | MMU/PTE faults and the fetch-line rule | engine copy times out with no semaphore; `GUEST CE COPY FAILED: scratch-PT pool exhausted ... capacity=512`; re-translated GPA yields a different HPA | The GMMU caches PTE fetches in 128-byte lines and there is no invalidate: a runtime PTE write into an already-fetched line is invisible. A guest page's HPA can change between translations. | `rm-call: guest HPA changed: gpa=... was ... now ...`; `work_guest` `pool_exhausted`/`next_line` | Always use a fresh never-reused line per pair; pre-build guest-window slots before engine access; pin translated pages; treat pool exhaustion as end-of-budget, not a bug. | | PBDMA method/class errors | `0x1006` with `intr0` nonzero (a method/class was rejected) versus `intr1` bit 31 (`CTXNOTVALID`); NOCAT `faultingEngine="GR"`; errString `"Channel ID ... intr0 ... intr1 ..."` | Wrong method encoding, class/subchannel or channel state; `intr0 != 0` means the engine rejected the method, so look at the pushed method block, not the context only. | `gsp-phys-status-drain`; compare the method encoding with the engine's class | Fix the method/class encoding; no retry against the same wedged channel; re-run from a fresh ladder. | | Relay symptoms | `GUEST CE RELAY FAILED: semaphore released ... does not match ... (retried after 2 ms)`; `verify_fail`/`verify_transient`/`hold_verify_fail`; `fallback`/`skipped_sw`; `pool_exhausted`; `guest HPA changed` | One persistent data mismatch (shadow-page/concurrent-guest race or decode offset edge) is the open residual; anything else is the designed fail-closed fallback to the software apply. | `work_relay` line; the relay failure log includes sem target/payload/poll, first differing byte, live/snapshot dumps, `src_mutated`, `snap_vs_dst` | Keep default off until the mismatch is triaged. Abort the feature (`gsp_phys_relay=0`, then `gsp_phys_work_forward=0`) if `timeouts>0` or `verify_fail>0` ever appears. | @@ -779,7 +779,7 @@ bump, device/ISA/ROM change, protected state, matrix/CI) are owned by | Failed mutating RPC wedges GSP-RM | One bad mutating operation (channel-flag incident; `PROMOTE_CTX` assert `0x52a8b04`) leaves the server unable to reply or suspend while WPR2 stays up. | One mutating attempt per staged step; never retry a failed mutating RPC; abort the ladder from the first failure; the release fail-safe keeps the locks and refuses the hand-back. | | debugfs write side effects | Every 0200 file performs a real RPC/device action when its gates hold; a "test" write changes device or guest state. | Root-only; read `gsp-phys-state` and record the intended effect before any write; one write per action; no trigger writes outside a supervised cycle. | | `pkill -f qemu-system-x86_64` self-match | The pattern also matches the invoking shell's own command line, killing the agent/session instead of the VM. | Use bracket patterns (`pgrep/pkill -f '[q]emu-system-x86_64'`) or explicit PIDs; stop VMs through `scripts/vgpu_vm.py`. | -| Root-only debugfs | The whole `open-iov-vfio` debugfs directory and its files are root-only; a non-root existence check reports absence. | Run existence/read checks under root; treat "missing" as a permission question first. | +| Root-only debugfs | The whole `vgsp` debugfs directory and its files are root-only; a non-root existence check reports absence. | Run existence/read checks under root; treat "missing" as a permission question first. | | Module reload/unload hazards | Kthread/workqueue teardown can hang; the IOMMU identity-domain property and the guest-RAM file must not outlive the session; unloading with a live mdev/VM corrupts the teardown order. | Remove mdevs and stop VMs first; `gsp_sema_service_work_quiesce()` before staging is freed (release does this); use `scripts/stack_manager.sh unload`/`cleanup-crash`; audit leftovers after every unload. | | pstore/journal persistence | A freeze leaves no oops but usually a persistent pstore record; the journal stops. | Check `/sys/fs/pstore` before and after every cycle; never clear it; capture it before any reboot or retry - it is the only post-freeze evidence. | | Guest-RAM shared-file requirement | The engine can reach guest pages only through the QEMU file-backed shared region; a non-shared or non-preallocated RAM backend has no host page cache to translate. | Start QEMU with `memory-backend-file,share=yes,prealloc=on`; configure the file via `gsp-phys-guest-ram`; one file per module cycle; translation is gated on an open mdev and an active range. | -- cgit v1.2.3