From c17da06d6cd8b9c69449904637bf37729a962989 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 11:28:49 -0400 Subject: docs: engineering-operations playbook and project-wide status accuracy pass - docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links). --- .../03-architecture/08-physical-gpu-integration.md | 91 ++++++++++++++-------- 1 file changed, 60 insertions(+), 31 deletions(-) (limited to 'docs/03-architecture/08-physical-gpu-integration.md') diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index 2ea44a1..532cd77 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -1,10 +1,12 @@ # Physical GPU Integration: How Work Reaches Real Hardware -> **Status: mechanism chosen; ownership + firmware staging implemented; -> power-domain lock verified; first real GSP boot and first live GSP-RM -> exchange complete (2026-10-01); host-side initialization follow-up and a -> clean release are in progress.** The control plane works today: the guest's -> driver initializes and the CUDA driver-API baseline passes end to end +> **Status: backend live; init-done, static configuration and clean releases +> proven; CE work forwarding with an engine-authored release and fault relay +> implemented default-off; GR execution is the open wall. The device is +> currently in the designed locked fail-safe awaiting a supervised reboot +> after a GR context-promote attempt wedged GSP-RM.** The control plane works +> today: the guest's driver initializes and the CUDA driver-API baseline +> passes end to end > (context creation, allocation, copies, 64/64 payload round-trip), and the > guest's decoded work is executed in software as a development bridge. This > document records the interfaces that actually exist, the mechanism options, @@ -145,8 +147,10 @@ re-bind. While the module owns the device it verifies that the GPU still answers config space before proceeding and aborts on any power-loss signature (the companion controller reporting a lost root hub, or a failed config read). The distribution driver is then rebound with the lock still held, its probe is -allowed to complete, and only then is runtime power management restored to -automatic. Any power-loss signature aborts the sequence and forbids a retry in +allowed to complete, and the D0/D3cold locks stay latched by default +(`gsp_phys_repower=0`); restoring runtime power management is an explicit +opt-in through the power-restore path, taken only when the operator chooses +it. Any power-loss signature aborts the sequence and forbids a retry in that boot: an out-of-band platform power cut bypasses the lock and would produce the identical signature, and retrying would be a second dice roll on a host that has already lost the board. @@ -210,19 +214,22 @@ bridge of the GPU's domain), built from the live PCI topology method-data block is sent invalid/zero: the boot path deliberately makes no ACPI/AML calls and every consumer tolerates the no-data encoding. The server waits for the host to execute the sequencer command buffer carried in that -event before it raises `GSP_INIT_DONE` (0x1001). **A plain release is not yet -possible:** the Booter Unload gate requires all four secure scratch14 handoff -bits (24–27); a -host-driven boot sets only bit 24, and bit 27 is set by the server's shutdown -path (host RPC `UNLOADING_GUEST_DRIVER`; implemented in the module as the -rung-4b command-queue TX + shutdown path and pending its first live run), -while the -host-side seed/unlock alternative is fuse-locked on this part. Until that RPC -has run in a live session, a green session is left claimed with the harness -`--keep` flag and returning the GPU to the distribution driver takes a power -cycle. The supervised harness modes are `--boot` (the only write mode: full -boot lifecycle), `--boot-dryrun` (zero-write staging rehearsal) and `--keep` -(leave a verified claimed session running); `--fwsec` is retired. +event before it raises `GSP_INIT_DONE` (0x1001). **Clean release is +proven:** since 2026-10-01 the unload RPC path delivers a clean release with +no reboot on every cycle — the +Booter Unload gate requires all four secure scratch14 handoff bits (24–27); a +host-driven boot sets bit 24, and the server's shutdown path (host RPC +`UNLOADING_GUEST_DRIVER`, rung 4b, §8) sets the rest, while the host-side +seed/unlock alternative is fuse-locked on this part. The exception is a wedged +server: if the server never sets its bit, teardown cannot clear the protected +region and the fail-safe refuses the hand-back (no rebind, D0 locks held, a +reboot with approval required; §7). The supervised harness modes are `--boot` +(the only write mode: full boot lifecycle), `--boot-dryrun` (zero-write +staging rehearsal) and `--keep` (leave a verified claimed session running); +`--fwsec` is retired. The full parameter, debugfs and runbook inventory for +this path is maintained in the engineering-operations document +([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate +those tables. ### 4.4 Host-side resource-manager client @@ -277,12 +284,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build | | Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | | Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) | -| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8) | -| Real GSP boot by the module | first live boot 2026-10-01 — FWSEC WPR2 up, Booter Load `M0=0`, RISC-V active; clean release pending the server shutdown RPC | -| Host-side resource-manager client for real objects/channels | transport alive 2026-10-01 — first real GSP-RM event (0x1002) received; init-done (0x1001) awaits host sequencer execution | -| Work translation and real submission | not started | -| Completion relay through real hardware | not started | -| CUDA kernel execution (`vectorAdd`) | not started | +| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) | +| Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 | +| Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine | +| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; GR execution open | +| Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage | +| CUDA kernel execution (`vectorAdd`) | blocked on GR context validity/execution — the host GR ladder's first push faulted `CTXNOTVALID` and the promote attempt wedged GSP-RM | ## 7. Open questions @@ -297,6 +304,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. controller action that does not go through the root port's D3 path — would produce the identical power-loss signature; the claim must abort on that signature and must not be retried in the same boot. +* **Wedged server / protected-region fail-safe:** if a live session wedges the + server so its shutdown path never completes, WPR2 cannot be cleared (the + host-side unlock is fuse-locked on this part). The release then refuses to + rebind and keeps the D0 locks; never retry in that boot, and return to the + distribution driver only through a reboot governed by the project's + reboot-approval policy (a written reason and post-reboot plan first). * **Supervised operation of the lock:** the lock is implemented and verified across three supervised claim/release cycles (2026-10-01). Every claim remains supervised: explicit human go-ahead, other host work warned, a @@ -338,7 +351,8 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`), and host-side seed/unlock is fuse-locked on this part. The live session is therefore left claimed (`--keep`) and a fresh cycle needs a power cycle. - See §4.3. + See §4.3. *(Historical: superseded by the clean-release result in the + unload-RPC update below.)* * **Update (2026-10-01) — rung 4b implemented (build-only):** the module now produces real command-queue records to GSP-RM and attempts a clean shutdown on release. The producer writes the 48-byte transport element header plus @@ -355,8 +369,10 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. behaviour is unchanged. A write-only debugfs trigger (`open-iov-vfio/gsp-phys-unload-rpc`, write "1") lets a live claimed session test the shutdown without touching WPR2. All of it is inert unless - the full boot stage ran; the code has not yet run live, and this bullet - must be updated with the live outcome. + the full boot stage ran. **Live-verified (2026-10-01):** the unload RPC + shuts the server down (GSP-RM processor suspended in ~220 ms, handoff bits + complete), so WPR2 goes down and the distribution driver is rebound with no + reboot on every clean cycle. * **Update (2026-10-01) — rung 4c slice 1 implemented (build-only):** for the next cycle a host resource-manager static-info probe is available: the write-only debugfs file `open-iov-vfio/gsp-phys-rm-probe` (write "1", only @@ -365,8 +381,20 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. handles, split-VA flag, BAR1/2 PDE bases, framebuffer geometry, engine caps), falling back once to `SET_GUEST_SYSTEM_INFO` plus one retry; the command-queue producer now splits records across up to 16 queue elements - and the status-queue walk coalesces multi-element records. Not yet run - live; the module was built only. + and the status-queue walk coalesces multi-element records. **Live-verified:** + init-done with the full 1656-byte static-config reply was observed on the + physical server, and the probe decodes its static configuration (internal + handles, BAR PDE bases and engine caps); the multi-element producer and + status-walk coalescing also run live. +* **Update (2026-10-02) — GR execution is the wall; device in locked + fail-safe:** the resource-manager client ladder reaches the compute engine + (GR0 channel plus a compute-class object), but the first GR semaphore push + faulted with an uncommitted context (`CTXNOTVALID`), and the follow-up + `GPU_PROMOTE_CTX` attempt wedged GSP-RM (internal assert `0x52a8b04`, no + reply). Teardown could not clear WPR2, so the fail-safe held the device + (module unloaded, GPU unbound, protected region up, D0 locks held). Recovery + is a supervised reboot under the reboot-approval policy; the promote + bisection resumes after that. ## 9. Interaction sequence (end state) @@ -398,6 +426,7 @@ system map is in [`09-full-system-picture.md`](09-full-system-picture.md). ## See also - [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts +- [`../eng-ops.md`](../eng-ops.md) - engineering operations; Part II carries the physical-backend parameter, debugfs and runbook inventory - [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans - [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs - [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view -- cgit v1.2.3