From c17da06d6cd8b9c69449904637bf37729a962989 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 11:28:49 -0400 Subject: docs: engineering-operations playbook and project-wide status accuracy pass - docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links). --- docs/03-architecture/01-system-overview.md | 12 ++- docs/03-architecture/02-guest-stack-and-cudasw.md | 15 +-- docs/03-architecture/03-gsp-and-gsp-rm.md | 12 ++- docs/03-architecture/04-uvm.md | 16 +-- docs/03-architecture/05-gmmu-amap-hal-gr.md | 3 +- .../06-host-module-and-hybrid-model.md | 37 +++++-- .../07-driver-lifecycle-field-guide.md | 10 +- .../03-architecture/08-physical-gpu-integration.md | 91 +++++++++++------ docs/03-architecture/09-full-system-picture.md | 38 ++++--- .../10-interaction-and-scheduling.md | 24 ++--- .../11-physical-backend-guest-wiring.md | 109 +++++++++++++++------ 11 files changed, 247 insertions(+), 120 deletions(-) (limited to 'docs/03-architecture') diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index 5c2ba6c..fa328ae 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -60,7 +60,7 @@ HOST MODULE open-iov-vfio.ko ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing - └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] + └─ work path: channel/push forwarding to the physical GPU [CE copies forwarded (default off); GR open] │ module-owned device + real GSP session HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module │ (the distribution nvidia.ko stays unmodified and unbound in @@ -170,10 +170,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and without server-side object management. Where the driver consumes a reply value, that value is still being corrected (the reply-fidelity ladder). 5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts - (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches - `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion - wait; the copy-engine method streams execute in software as a bridge, and - the execution backend is not built yet (see + (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed); context creation + completes and the full CUDA driver-API smoke passes (`ALL OK` 64/64, copies + matched). The execution backend is live: CE-class copies are + forwarded to the physical copy engine (default off) with engine-authored + semaphore releases and a fault relay; kernel execution is blocked on GR + engine-context validity (see [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). The CUDA-side contract and the reply-fidelity rules are documented in diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index e841342..f9fe249 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -88,14 +88,15 @@ open /dev/nvidia* → cuInit → UVM registration (per process) ``` The initialization path is where compatibility is won or lost; the runtime -path is where *execution* is won or lost. Open-IOV's current focus is -pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client -reaches context creation, which currently ends in a forced `rc=999` on a -completion wait; the engine inventory is complete and the forwarding decision -is made — a module-owned GPU running its real GSP firmware as the -resource-manager server (see +path is where *execution* is won or lost. Context creation and the full CUDA +driver-API smoke now pass (`ALL OK`, copies matched), so the current focus is +execution: the forwarding decision is made and the backend is live — a +module-owned GPU running its real GSP firmware as the resource-manager server +(see [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) -and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). +and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) — with +CE-class copies forwarded to the physical copy engine (default off) and GR +kernel execution blocked on GR engine-context validity. For the CUDA userspace side of this stack — its architecture, how it is built and executes, and the exact kernel/GPU interface contract we must satisfy — diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 7f206ba..10e6a73 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -59,8 +59,10 @@ firmware: FWSEC raised WPR2 with a zero FRTS error code, the SEC2 booter halted with mailbox0 = 0, the RISC-V core reported active (`RISCV_STATUS=0x00000035`, active bit 0) and `FALCON_OS` read 0. The server's first message was then a status-queue event, function `0x1002` (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, -`rpc_result=0`), carrying a command buffer the host must execute; -`GSP_INIT_DONE` (`0x1001`) follows only after that sequencer runs. The +`rpc_result=0`), carrying a command buffer the host must execute. Once that +sequencer runs, `GSP_INIT_DONE` (`0x1001`) is delivered with `rpc_result=0`, +the static-configuration reply is served from the observed device tables, and +the session remains live until the unload RPC releases it at teardown. The host-side queue pair behind the exchange is one shared page table plus a 256 KB command queue and a 256 KB status queue of 4 KB elements. @@ -229,8 +231,10 @@ is not lost. The engine-inventory row was the CUDA blocker at the time of writing: `libcuda` filters the engine list it receives against the class list and expects the async copy engine to be present, because the secondary (async) -engine record is otherwise zeroed. That inventory is now complete; the -context-creation wait is later, in the completion path. See +engine record is otherwise zeroed. That inventory is now complete and context +creation passes; the current frontier is engine execution, where CE-class +copies are forwarded to the physical copy engine (default off) and GR kernel +execution is blocked on GR engine-context validity. See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the offsets and the consumed fields, and [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 4c8a0c9..9396e6a 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,17 +50,19 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, the inventory now includes the async copy engine, and CE copies are forwarded to the physical copy engine (default off) with engine-authored releases | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); no GR kernel is forwarded yet (blocked on GR engine-context validity) | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | -| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | +| Fault delivery | fault buffer + notification; replay/cancel semantics | 🟡 backend-side fault observation/relay exists (default off: notifier write + event post); the guest replay/cancel contract is open | | Semantics of completions | tracker values must advance exactly | 🟡 | -**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and -currently ends in a forced `rc=999` on a completion wait; the engine inventory -is complete, and the remaining work is completion translation plus the chosen -execution backend +**Current frontier (2026-10-02):** context creation and the full CUDA +driver-API smoke pass (`ALL OK`, copies matched); the execution backend is +live and CE-class copies are forwarded to the physical copy engine (default +off) with engine-authored releases and a fault relay. GR kernel execution is +blocked on GR engine-context validity; the remaining UVM work is completion +translation and the guest-side fault replay/cancel contract ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). Failed-context teardown has produced guest kernel oopses in `uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index 76d4200..67ceeca 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -206,5 +206,4 @@ section 3.7 for the record layout the driver's engine walker consumes. | Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` | **Rule:** tracked documentation describes components and behaviour, with our -own observations, IDs and layouts. Exact analysis sources live in the -maintainers' study notes. +own observations, IDs and layouts. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index d6b96e7..0dbc89a 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -140,7 +140,10 @@ section 3.7 for the analysis. method streams and executes the work it understands in software - memsets, copies and semaphore releases - and completes the corresponding semaphores. That is a correctness bridge for memory operations, not the data plane: no -CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend +CUDA kernel is forwarded to the physical GPU yet (default-off CE copy +forwarding, engine-authored release relay and fault relay are implemented; see +[`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)); +the chosen execution backend (a module-owned GPU running its real GSP firmware as the resource-manager server) is in [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the @@ -149,10 +152,11 @@ trade-offs in section 3. **Subsystem status note.** Some components are compiled but not currently -wired into the live path: the physical-GSP load/boot path is skipped in favour -of emulation, and the runtime firmware analyzer, version tables, hardware -abstraction tables and dynamic RPC dispatcher are not yet called by the live -flow. Treat their descriptions here as intended design, not active code. +wired into the live path: the physical-GSP load/boot path is default-off and +runs only behind the supervised claim gates, and the runtime firmware +analyzer, version tables, hardware abstraction tables and dynamic RPC +dispatcher are not yet called by the live flow. Treat their descriptions here +as intended design, not active code. ### 3.3 Event channels and interrupts @@ -263,6 +267,22 @@ emulated device inspectable without rebuilding: | `sema_map_add=,` | teach the service one verified `{GPU VA -> guest PA}` pair | | `dump_guest=,` | dump guest physical memory (length capped) | +The physical-backend path adds a second, default-off parameter set: +`gsp_phys_gpu`, `gsp_phys_hazard_ack`, `gsp_phys_fw`, `gsp_phys_mdev_owned`, +`gsp_phys_static_live`, `gsp_phys_ctrl_forward`, `gsp_phys_work_forward`, +`gsp_phys_guest_mem`, `gsp_phys_work_defer_release`, +`gsp_phys_work_hw_primary`, `gsp_phys_relay`, `gsp_phys_fault_relay`, +`gsp_phys_fault_inject`, `gsp_phys_rmcall_engine`, +`gsp_phys_rmcall_gr_promote`, `gsp_phys_repower`, `gsp_phys_layout`, +`gsp_phys_boot`, `gsp_phys_boot_dryrun`, `gsp_phys_nocat` and the +`gsp_phys_fwsec*` stage controls. Their semantics, the physical-backend +debugfs inventory (`gsp-phys-state`, `gsp-phys-rm-probe`, `gsp-phys-rm-call`, +`gsp-phys-unload-rpc`, `gsp-phys-guest-ram`/`gsp-phys-guest-gpa`, +`gsp-phys-log-dump`, `gsp-phys-status-drain`) and the supervised runbooks are +maintained in the engineering-operations document +([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate +those tables. + These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a mismatch. Their workflows are documented in @@ -312,8 +332,9 @@ in `kernel-open` first). | Message queues + RPC + all init controls observed from stock 570.144 | implemented | | Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | | Push decoding + semaphore completion | demonstrated live; being productized | -| Page-table population for server-owned ranges | designed, not implemented | -| GPU work forwarding (GR/CE execution) | designed, not implemented | -| Execution backend (module-owned GPU + direct real-GSP session) | mechanism chosen; ownership + firmware staging verified live; power-domain lock implemented, pending supervised verification; real GSP boot not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | +| Guest-RAM physical path | implemented default-off: guest RAM is file-backed shared memory and GPA→HPA goes through its page cache, verified live on the CE | +| Page-table population for server-owned ranges | implemented default-off: forwarded CE copies map resolved host pages through module-built GMMU scratch page tables; real guest page tables remain | +| GPU work forwarding (GR/CE execution) | CE forwarding, engine-authored release relay and fault relay implemented default-off and verified live; GR execution open | +| Execution backend (module-owned GPU + direct real-GSP session) | live: boot, init-done, full static-config reply and clean releases proven; RM client allocates objects/channels and has executed real CE copies; currently in the designed locked fail-safe after a GR context-promote wedge ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented | diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 1bfe912..73b007e 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -107,7 +107,7 @@ fallback). Representative controls: | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | | `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path | | `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | -| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | +| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | historically issued immediately before context creation and answered by the generic fallback. Context creation now returns; in the host GR ladder this promote is the load-bearing GR context-validity operation, and a wrong shape is a known server wedge (no retry; reboot policy - [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7) | | `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | | `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently | @@ -177,6 +177,14 @@ start at the execution backend is in [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). +**Update (2026-10-02):** context creation now returns (the driver-API smoke +completes end to end) and the frontier is execution. The host GR ladder +allocates a GR0 channel and a compute-class object, but its first semaphore +push faults with an uncommitted GR context (`CTXNOTVALID`), and the +`GPU_PROMOTE_CTX` attempt that followed wedged GSP-RM; the device then sits in +the designed locked fail-safe (see +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7/§8). + **Log signatures (guest side):** ``` nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index 2ea44a1..532cd77 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -1,10 +1,12 @@ # Physical GPU Integration: How Work Reaches Real Hardware -> **Status: mechanism chosen; ownership + firmware staging implemented; -> power-domain lock verified; first real GSP boot and first live GSP-RM -> exchange complete (2026-10-01); host-side initialization follow-up and a -> clean release are in progress.** The control plane works today: the guest's -> driver initializes and the CUDA driver-API baseline passes end to end +> **Status: backend live; init-done, static configuration and clean releases +> proven; CE work forwarding with an engine-authored release and fault relay +> implemented default-off; GR execution is the open wall. The device is +> currently in the designed locked fail-safe awaiting a supervised reboot +> after a GR context-promote attempt wedged GSP-RM.** The control plane works +> today: the guest's driver initializes and the CUDA driver-API baseline +> passes end to end > (context creation, allocation, copies, 64/64 payload round-trip), and the > guest's decoded work is executed in software as a development bridge. This > document records the interfaces that actually exist, the mechanism options, @@ -145,8 +147,10 @@ re-bind. While the module owns the device it verifies that the GPU still answers config space before proceeding and aborts on any power-loss signature (the companion controller reporting a lost root hub, or a failed config read). The distribution driver is then rebound with the lock still held, its probe is -allowed to complete, and only then is runtime power management restored to -automatic. Any power-loss signature aborts the sequence and forbids a retry in +allowed to complete, and the D0/D3cold locks stay latched by default +(`gsp_phys_repower=0`); restoring runtime power management is an explicit +opt-in through the power-restore path, taken only when the operator chooses +it. Any power-loss signature aborts the sequence and forbids a retry in that boot: an out-of-band platform power cut bypasses the lock and would produce the identical signature, and retrying would be a second dice roll on a host that has already lost the board. @@ -210,19 +214,22 @@ bridge of the GPU's domain), built from the live PCI topology method-data block is sent invalid/zero: the boot path deliberately makes no ACPI/AML calls and every consumer tolerates the no-data encoding. The server waits for the host to execute the sequencer command buffer carried in that -event before it raises `GSP_INIT_DONE` (0x1001). **A plain release is not yet -possible:** the Booter Unload gate requires all four secure scratch14 handoff -bits (24–27); a -host-driven boot sets only bit 24, and bit 27 is set by the server's shutdown -path (host RPC `UNLOADING_GUEST_DRIVER`; implemented in the module as the -rung-4b command-queue TX + shutdown path and pending its first live run), -while the -host-side seed/unlock alternative is fuse-locked on this part. Until that RPC -has run in a live session, a green session is left claimed with the harness -`--keep` flag and returning the GPU to the distribution driver takes a power -cycle. The supervised harness modes are `--boot` (the only write mode: full -boot lifecycle), `--boot-dryrun` (zero-write staging rehearsal) and `--keep` -(leave a verified claimed session running); `--fwsec` is retired. +event before it raises `GSP_INIT_DONE` (0x1001). **Clean release is +proven:** since 2026-10-01 the unload RPC path delivers a clean release with +no reboot on every cycle — the +Booter Unload gate requires all four secure scratch14 handoff bits (24–27); a +host-driven boot sets bit 24, and the server's shutdown path (host RPC +`UNLOADING_GUEST_DRIVER`, rung 4b, §8) sets the rest, while the host-side +seed/unlock alternative is fuse-locked on this part. The exception is a wedged +server: if the server never sets its bit, teardown cannot clear the protected +region and the fail-safe refuses the hand-back (no rebind, D0 locks held, a +reboot with approval required; §7). The supervised harness modes are `--boot` +(the only write mode: full boot lifecycle), `--boot-dryrun` (zero-write +staging rehearsal) and `--keep` (leave a verified claimed session running); +`--fwsec` is retired. The full parameter, debugfs and runbook inventory for +this path is maintained in the engineering-operations document +([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate +those tables. ### 4.4 Host-side resource-manager client @@ -277,12 +284,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build | | Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | | Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) | -| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8) | -| Real GSP boot by the module | first live boot 2026-10-01 — FWSEC WPR2 up, Booter Load `M0=0`, RISC-V active; clean release pending the server shutdown RPC | -| Host-side resource-manager client for real objects/channels | transport alive 2026-10-01 — first real GSP-RM event (0x1002) received; init-done (0x1001) awaits host sequencer execution | -| Work translation and real submission | not started | -| Completion relay through real hardware | not started | -| CUDA kernel execution (`vectorAdd`) | not started | +| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) | +| Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 | +| Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine | +| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; GR execution open | +| Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage | +| CUDA kernel execution (`vectorAdd`) | blocked on GR context validity/execution — the host GR ladder's first push faulted `CTXNOTVALID` and the promote attempt wedged GSP-RM | ## 7. Open questions @@ -297,6 +304,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. controller action that does not go through the root port's D3 path — would produce the identical power-loss signature; the claim must abort on that signature and must not be retried in the same boot. +* **Wedged server / protected-region fail-safe:** if a live session wedges the + server so its shutdown path never completes, WPR2 cannot be cleared (the + host-side unlock is fuse-locked on this part). The release then refuses to + rebind and keeps the D0 locks; never retry in that boot, and return to the + distribution driver only through a reboot governed by the project's + reboot-approval policy (a written reason and post-reboot plan first). * **Supervised operation of the lock:** the lock is implemented and verified across three supervised claim/release cycles (2026-10-01). Every claim remains supervised: explicit human go-ahead, other host work warned, a @@ -338,7 +351,8 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`), and host-side seed/unlock is fuse-locked on this part. The live session is therefore left claimed (`--keep`) and a fresh cycle needs a power cycle. - See §4.3. + See §4.3. *(Historical: superseded by the clean-release result in the + unload-RPC update below.)* * **Update (2026-10-01) — rung 4b implemented (build-only):** the module now produces real command-queue records to GSP-RM and attempts a clean shutdown on release. The producer writes the 48-byte transport element header plus @@ -355,8 +369,10 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. behaviour is unchanged. A write-only debugfs trigger (`open-iov-vfio/gsp-phys-unload-rpc`, write "1") lets a live claimed session test the shutdown without touching WPR2. All of it is inert unless - the full boot stage ran; the code has not yet run live, and this bullet - must be updated with the live outcome. + the full boot stage ran. **Live-verified (2026-10-01):** the unload RPC + shuts the server down (GSP-RM processor suspended in ~220 ms, handoff bits + complete), so WPR2 goes down and the distribution driver is rebound with no + reboot on every clean cycle. * **Update (2026-10-01) — rung 4c slice 1 implemented (build-only):** for the next cycle a host resource-manager static-info probe is available: the write-only debugfs file `open-iov-vfio/gsp-phys-rm-probe` (write "1", only @@ -365,8 +381,20 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. handles, split-VA flag, BAR1/2 PDE bases, framebuffer geometry, engine caps), falling back once to `SET_GUEST_SYSTEM_INFO` plus one retry; the command-queue producer now splits records across up to 16 queue elements - and the status-queue walk coalesces multi-element records. Not yet run - live; the module was built only. + and the status-queue walk coalesces multi-element records. **Live-verified:** + init-done with the full 1656-byte static-config reply was observed on the + physical server, and the probe decodes its static configuration (internal + handles, BAR PDE bases and engine caps); the multi-element producer and + status-walk coalescing also run live. +* **Update (2026-10-02) — GR execution is the wall; device in locked + fail-safe:** the resource-manager client ladder reaches the compute engine + (GR0 channel plus a compute-class object), but the first GR semaphore push + faulted with an uncommitted context (`CTXNOTVALID`), and the follow-up + `GPU_PROMOTE_CTX` attempt wedged GSP-RM (internal assert `0x52a8b04`, no + reply). Teardown could not clear WPR2, so the fail-safe held the device + (module unloaded, GPU unbound, protected region up, D0 locks held). Recovery + is a supervised reboot under the reboot-approval policy; the promote + bisection resumes after that. ## 9. Interaction sequence (end state) @@ -398,6 +426,7 @@ system map is in [`09-full-system-picture.md`](09-full-system-picture.md). ## See also - [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts +- [`../eng-ops.md`](../eng-ops.md) - engineering operations; Part II carries the physical-backend parameter, debugfs and runbook inventory - [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans - [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs - [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view diff --git a/docs/03-architecture/09-full-system-picture.md b/docs/03-architecture/09-full-system-picture.md index e03a68e..3c15830 100644 --- a/docs/03-architecture/09-full-system-picture.md +++ b/docs/03-architecture/09-full-system-picture.md @@ -80,7 +80,7 @@ flowchart TB class VDEV,EMU done class MEMM,WORK wip - class BE plan + class BE done class IOMMU,VFIO done class GSP,ENG,GMMU,VRAM,DB hw class ND,USR,GKO,APP vendor @@ -125,8 +125,9 @@ stateDiagram-v2 Emulated today: the module drives the stage transitions and answers status reads from the BAR0 shadow. The real - backend boots the physical GSP in - parallel [planned]. + backend boots the physical GSP as its + own live session (init-done, static + configuration and clean releases proven). end note ``` @@ -262,14 +263,14 @@ flowchart LR F --> G["Write releases and completions
into guest memory"] G --> H["Guest trackers advance"] - D -.->|"planned"| I["Forward to a real engine"] + D -.->|"wip/default-off"| I["Forward to a real engine"] classDef done fill:#d5f5d5,stroke:#1e7a1e classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a class A,B,D done class E,F,G,H wip - class I plan + class I wip ``` This is a correctness bridge: it advances synchronization and memory @@ -329,22 +330,30 @@ flowchart TB R4 --> R5["Real submissions scheduled by GSP"] R5 --> R6["Completion and faults relayed to guest"] end - G -.->|"translation [planned]"| R4 + G -.->|"translation [wip/default-off]"| R4 classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a class A,B,C,D,E done class F,G wip - class R1,R2,R3,R4,R5,R6 plan + class R1,R2,R3 done + class R4,R5,R6 wip ``` +Backend boot, the real RM client, channel allocation and real CE copies are +live; guest-forwarding and relay are default-off; GR execution and real guest +page tables are the open `[planned]` parts (see +[`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)). + ## 5. Scheduling and multi-tenancy There is no hardware partitioning (no SR-IOV) on this GPU generation, so tenancy is a **logical VF model**: each VM holds one mdev instance, and the module gives each instance its own GSP-RM session, channels, page tables and scheduling policy. “Per-VF GSP emulation” therefore means per-mdev emulation -context plus a per-mdev real GSP client in the backend. +context plus a per-mdev real GSP client in the backend. Today the backend is +a single per-GPU session (live); per-mdev clients are planned. ```mermaid flowchart TB @@ -358,7 +367,7 @@ flowchart TB Q2["Emulation context 2"] Q3["Emulation context N"] POL["Fairness policy [planned]
time slices, quotas, priority, burst"] - C1["Real GSP client 1 [planned]"] + C1["Real GSP client 1
single per-GPU session [live]"] C2["Real GSP client 2 [planned]"] C3["Real GSP client N [planned]"] end @@ -381,7 +390,8 @@ flowchart TB classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a class Q1,Q2,Q3 done - class C1,C2,C3,POL,SCHED plan + class C1 done + class C2,C3,POL,SCHED plan ``` **How multi-tenant performance works (and is intended to):** @@ -636,10 +646,10 @@ against a stock guest driver with evidence in the repository. | Guest memory access + VA-to-PA discovery | partial | yes | protocol-derived discovery, no external pins | | Push decode + semaphore completion | partial | yes | complete GR/compute encodings | | Software execution bridge (memsets, copies, releases) | partial | yes | remains bridge only | -| Page-table population for server-owned ranges | no | no | design + validator, then implement | -| Execution backend (own GPU + real GSP) | no | no | device ownership, real GSP bring-up, RM client | -| Real work forwarding (GR and CE) | no | no | after backend bring-up | -| Fault semantics (buffers, replay, cancel) | no | no | after forwarding | +| Page-table population for server-owned ranges | partial (module-built scratch page tables, default off) | yes (default off) | real guest page tables | +| Execution backend (own GPU + real GSP) | yes (single per-GPU session) | yes | per-mdev clients | +| Real work forwarding (GR and CE) | partial (CE default-off) | yes (default off) | GR context validity/execution | +| Fault semantics (buffers, replay, cancel) | partial (forwarded-engine observation/relay default-off) | yes (default off) | GR fault semantics and triage | | Multi-tenant fairness policy | no | no | after multi-tenant forwarding | | Version matrix (2+ driver families) | partial | partial | per-version profiles + A/B | | Windows guest | no | no | after Linux execution path | diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md index 3608dc5..ffc02d7 100644 --- a/docs/03-architecture/10-interaction-and-scheduling.md +++ b/docs/03-architecture/10-interaction-and-scheduling.md @@ -22,12 +22,12 @@ to, and where it stands. | Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] | | Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] | | RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] | -| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] | -| Semaphore release to guest memory | module writes the payload word into the resolved guest page | us | [done] | +| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR execution open) | +| Semaphore release to guest memory | module writes the payload word into the resolved guest page; a forwarded CE copy can have the engine author it | us | [done] (relay default off) | | VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] | | Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] | -| Real channel submission | translated push stream on a real channel, real doorbell | us + hardware | [planned] | -| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers | us | [planned] | +| Real channel submission | translated push stream on a real channel, real doorbell; the ladder's real submission is live | us + hardware | [wip] (guest forwarding default-off) | +| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers; forwarded-engine fault observation/relay exists | us | [wip] (default off) | | mdev create / destroy | sysfs plus VFIO device registration | ms | [done] | | Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] | | Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] | @@ -129,7 +129,7 @@ stateDiagram-v2 note right of Applying State and notification operations are applied from the emulation. Real engine - work is forwarded [planned]. + work is forwarded [wip/default-off]. end note ``` @@ -198,7 +198,7 @@ Rules the design enforces: | Push entry decode | us per entry | thousands of entries per context | | Rescue of an unreadable entry | bounded ms | only on translation misses | | MSI-X to guest | us | completion latency floor | -| Real submission + completion relay | hardware + us | the forwarded-work path [planned] | +| Real submission + completion relay | hardware + us | the forwarded-work path [wip/default-off] | The rule of thumb the architecture is built around: **state operations are cheaper in the module than they would be on hardware; real work belongs on @@ -250,7 +250,7 @@ flowchart TB F --> G["KVM injects MSI-X vector"] G --> H["Guest ISR runs"] end - subgraph PHY["Module and physical GPU [planned]"] + subgraph PHY["Module and physical GPU [wip/default-off]"] I["Real channel doorbell"] --> J["GPU executes"] J --> K["Physical MSI-X"] K --> L["Module IRQ handler"] @@ -260,9 +260,9 @@ flowchart TB M -.-> E classDef done fill:#d5f5d5,stroke:#1e7a1e - classDef plan fill:#f8d7da,stroke:#9a1a1a + classDef wip fill:#fff3cd,stroke:#9a7b1a class A,B,C,D,E,F,G,H done - class I,J,K,L,M plan + class I,J,K,L,M wip ``` ## 9. Lifecycle and teardown ordering @@ -301,8 +301,8 @@ sequenceDiagram participant GD as Guest driver participant EMU as GSP-RM emulation participant VT as Version profile - participant BE as Execution backend [planned] - participant FW as Real GSP firmware [planned] + participant BE as Execution backend (single live session) + participant FW as Real GSP firmware (live session) GD->>EMU: version and capability queries EMU->>VT: select profile for this driver family @@ -327,7 +327,7 @@ sequenceDiagram | Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] | | Pending release queue | releases awaiting a translation | decoder | service drain | [done] | | Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] | -| Real channels and page tables | forwarding state | backend | real GPU | [planned] | +| Real channels and page tables | forwarding state (real channel live; guest page tables open) | backend | real GPU | [wip] | ## See also diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md index cbb4b33..2229365 100644 --- a/docs/03-architecture/11-physical-backend-guest-wiring.md +++ b/docs/03-architecture/11-physical-backend-guest-wiring.md @@ -1,8 +1,11 @@ # Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session -> **Status: P0-P5(w1) implemented; P6a guest-page reachability and P6b waves +> **Status: P0-P5(w1) implemented; P6a guest-page reachability (trigger +> residual fixed) and P6b waves > 1/1.5/2 (guest-copy forwarding, deferred release, hardware-primary copies) -> proven live (2026-10-02).** The backend API skeleton +> proven live (2026-10-02); P6c C0/C1 (engine-authored guest CE semaphore +> relay) and C3 (forwarded-engine fault observation/relay) implemented +> default-off; H1 host GR ladder open.** The backend API skeleton > (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that > path is an open defect off the critical path), the static-config identity > overlay (P2, verified live in an A/B run), the read-only control @@ -16,7 +19,14 @@ > releases exactly once) and the wave-2 hardware-primary copies (default off, > verified live: copy_skipped=119, queued req=100 ok=100 verify_ok=100 > verify_fail=0 fallback=0 timeouts=0, smoke `matched=64/64` `ALL OK`, with -> the software fallback for unqueued/failed copies) are in the tree. +> the software fallback for unqueued/failed copies) are in the tree. The P6c +> relay is live-verified default-off — the physical engine authored the +> guest CE copy's semaphore release (`relays=64`), failed/refused copies fall +> back to the software apply exactly once, and one relayed-copy data mismatch +> is under triage (mismatch logging plus software fallback in charge); the C3 +> forwarded-engine fault relay is live green, default off. The P6a trigger +> residual is fixed: triggers 6/7 re-translate at trigger time and use fresh +> scratch lines. > The physical backend itself is live: the module-owned > GSP-RM completes initialization on the real TU104, serves the full > static-config reply, and shuts down gracefully. The host resource-manager @@ -24,7 +34,12 @@ > a committed page directory, a channel group and a real copy-engine channel, > and has executed the **first real work on the physical engine** — a 4 KiB > copy in virtual-address mode with the semaphore released and destination -> equal to source (cycle 25). What this document specifies is the remaining +> equal to source (cycle 25). The H1 host GR ladder then allocated a GR0 +> channel with a compute-class object, but its first semaphore push faulted +> with an uncommitted context (`CTXNOTVALID`), and the follow-up +> `GPU_PROMOTE_CTX` attempt wedged GSP-RM; the GR execution wall is open and +> the device sits in the designed locked fail-safe. What this document +> specifies is the remaining > bridge: wiring that live session into the guest-facing mdev/VFIO path so that > guest submissions execute on the physical GPU. It extends > [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which @@ -102,6 +117,15 @@ (`:5068`). Stage 4 now runs end to end: the channel allocates, the page directory is committed, and a 4 KiB virtual-address-mode copy executes on the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25). + `gsp_phys_rmcall_engine=1` selects a GR0 ladder variant (stage 3 builds the + channel and a `TURING_COMPUTE_A` compute object with `engineType=1`, stage 4 + pushes a GR semaphore release). Its first live push faulted with an + uncommitted context (`CTXNOTVALID`). `gsp_phys_rmcall_gr_promote=1..3` + selects opt-in `GPU_PROMOTE_CTX` shapes to commit that context; the first + attempt wedged GSP-RM (internal assert, no reply) and teardown could not + clear WPR2, so the fail-safe held the device. Debugfs trigger 9 re-runs the + GR semaphore push on an already-built GR ladder without touching WPR2, for + use once a promote shape is viable. * Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2 (`:911`), frees staging (`:914`), and only rebinds the distribution driver @@ -114,10 +138,12 @@ The two planes now meet at the lifecycle seam (P0/P1, mdev-owned claim/release), the static-config identity overlay (P2), the read-only control whitelist (P3) and, for eligible CE copies, the P6b scratch-mapped work path (waves 1/1.5/2: guest operands, deferred exactly-once release, -hardware-primary copies); objects, FB geometry, engine caps, real guest -page tables and completion/fault relay are still emulated. The remaining -bridge must be added at the reply point `gsp_rm_send()`, the inbound dispatch -point `gsp_rm_handle_element()`, and the work-plane seams (see §2). +hardware-primary copies) plus the P6c relay (C0/C1 engine-authored guest +release, C3 forwarded-engine fault observation/relay), all default off; +objects, FB geometry, engine caps and real guest page tables are still +emulated. The remaining bridge must be added at the reply point +`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the +work-plane seams (see §2). --- @@ -141,7 +167,9 @@ point `gsp_rm_handle_element()`, and the work-plane seams (see §2). BACKEND_STOPPING stop emulation forwarding; UNLOADING_GUEST_DRIVER; | quiesce; WPR2 teardown; staging free v - BACKEND_OFF power unlock; distribution driver rebound and settled + BACKEND_OFF power unlock only when gsp_phys_repower=1 (default + 0 latches the D0/D3cold locks); distribution driver + rebound and settled ``` `BACKEND_READY` is the single gate for every live answer. In every other @@ -198,8 +226,9 @@ generation counter` so a release cannot deadlock against a forward in flight. parent/class teardown. * **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out, the existing release still quiesces, runs the FWSEC teardown and refuses to - rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit - human approval plus a handoff update. + rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then follows the + project reboot-approval policy (a written reason and post-reboot plan before + approval). * **Static-config served before READY:** bounded wait (2 s, implemented) then pure emulation fallback, with a stat counter; never block the emulation thread longer than the guest's own RPC poll can tolerate. @@ -495,8 +524,10 @@ go-ahead, recovery plan, pstore, never unattended 514 real CE copies (194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts). The P6b waves 1/1.5/2 (guest-copy forwarding, deferred exactly-once release, hardware-primary copies with a software - fallback) have since landed (P6 status below); the real guest-page-table - path (P6c) remains. + fallback) have since landed (P6 status below), and P5 wave 2 (hardware- + primary release relay) is subsumed by the P6b wave-2 hardware-primary mode + and the P6c engine-authored relay; the real guest-page-table path and GR + execution remain. * **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25: channel allocates, page directory commits, `REAL COPY OK` on the physical CE in VA mode). The remaining work: bind each guest CE/GR channel to a real @@ -587,20 +618,35 @@ go-ahead, recovery plan, pstore, never unattended with the guest-RAM ranges unregistered proved that fallback again (`copy_skipped=121`, recovered once each by the software copy, `copy_fail=0`, smoke still green), so the software fallback remains for any - unqueued or failed copy. **Residual (P6a trigger path):** - the debug-trigger path pre-builds its page-table slots once during ladder - stage 3; a guest page's host physical address can change between - translations (observed `0x13a591000` then `0x3b8318000` for the same GPA), - and a runtime PTE update into an already-fetched 128-byte line is invisible, - so the later write-direction trigger failed its data check. Forwarding is - unaffected (it resolves HPAs at submit time); the follow-up is to give the - debug triggers fresh scratch lines (the forwarding path) or to pin the - translated page. **Remaining:** P6c - completion/fault relay, real - page-directory shadowing, real guest channels and push forwarding with - guest page tables in the real MMU, and `vectorAdd` correct results; - guest-submission forwarding through the real guest page tables is not - implemented yet. All new parameters default off; with them off the software - path is unchanged. + unqueued or failed copy. **P6a trigger residual — fixed (2026-10-02):** + the debug triggers now re-translate the guest page's host physical address + at trigger time through the fresh-scratch-line path (triggers 6/7), retiring + the stale-PTE data-check failure; live `GUEST G2H COPY OK` and + `GUEST W2G COPY OK verify_ok=1`, with the PTE self-test and ladder + `REAL COPY OK` on a clean release. Forwarding was never affected (HPAs are + resolved at submit time). **P6c C0/C1 (engine-authored guest CE semaphore + relay, default off):** with `gsp_phys_relay=1` the guest-visible semaphore + page is mapped into the CE channel and the engine authors one selected + release; the deferred hold verifies the guest word and skips the software + apply of that record, and any refusal, failure, timeout or failed + verification leaves the unchanged software apply in charge, so the release + stays exactly-once. Live: the engine-authored release was observed in the + guest page (`relays=64`), the relay-on run stayed green + (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64), and a + forced-unresolvable control proved the fallback recovers every copy once; + one relayed copy released its semaphore but failed the host-side data + compare, so the parameter stays default-off pending triage, with the + mismatch logged and the software apply still in charge. **P6c C3 + (forwarded-engine fault observation/relay, default off):** with + `gsp_phys_fault_relay=1` a timed-out forwarded copy drains the GSP-RM + status queue for the engine's fault record, matches it to the ladder + channel and writes the guest channel notifier/event through the existing + MSI-X path (live green, default off). **Remaining:** real guest page tables + and channels, real page-directory shadowing, GR context validity/execution + and `vectorAdd` correct results; guest-submission forwarding through the + real guest page tables is not implemented yet, and the GR promote attempt + wedged the server (§0.2). All new parameters default off; with them off the + software path is unchanged. * **Scope:** map guest page-table pages/records so the physical GPU can DMA them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real server; allocate real memory objects for guest allocations (fn4 with real @@ -677,13 +723,18 @@ go-ahead, recovery plan, pstore, never unattended order). 8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only - `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree - (`backups/evidence-*`) current on every live cycle. + `gsp-phys-state`/`gsp-phys-log-dump` files, and attach the evidence + artifacts to the change on every live cycle. --- ## Appendix A - line anchor map (function -> file:line) +> **Historical anchors.** The line numbers below were taken from the +> working-tree revision used for the original design and are stale; they are +> kept as a historical map only. Anchor by function name when re-grepping and +> treat the numbers as untrusted. + `kernel/gsp_phys_booter.c` (working-tree revision used for this design): `gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init` 2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670, -- cgit v1.2.3