diff options
Diffstat (limited to 'docs/03-architecture')
| -rw-r--r-- | docs/03-architecture/13-vexecution-plane.md | 154 |
1 files changed, 92 insertions, 62 deletions
diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index 3cc6adb..733da13 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -16,7 +16,7 @@ > machine; read [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) > for the physical-session lifecycle this document builds on. > -> **Status: 2026-10-08, tree `master @ 66388560`.** Every empirical claim below +> **Status: 2026-10-08, tree `master @ 1f571dda`.** Every empirical claim below > cites a commit, a code location, a tracked document, or a local campaign > evidence file (paths under `backups/evidence-…/` are local evidence > artifacts, not shipped in the repository). Where a mechanism is a leading @@ -286,7 +286,7 @@ cuLaunchKernel | 7 | **Inline-QMD install / engine reads the descriptor** | **`[wall]` not observed** | | 8 | Context switch + SASS dispatch to SMs | `[wall]` not observed for forwarded launches | | 9 | Engine-authored release + channel release | `[wall]` absent for forwarded launches | -| 10 | Completion application, counter publish, notifier, event, MSI-X | `[wip]` CE path live default-off; GR wake-up is the open question (ISSUE-0001) | +| 10 | Completion application, counter publish, notifier, event, MSI-X | `[wip]` CE path live default-off; for GR the counter-slot publication and the GP_GET write-back are the open question (ISSUE-0001, FORENSICS-grv5) | The stage table's authority is [`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) @@ -294,7 +294,9 @@ The stage table's authority is re-verified by the `grv3` campaign: ring entries *were* consumed for ranks 0–2, releases *were* applied, and the guest still did not proceed (local evidence `backups/evidence-20261002-victory/analysis-2026-10-08-stall-v2/FORENSICS-grv3.md` -§0 and §2). +§0 and §2). The later `grv5` campaign (same evidence directory, +`FORENSICS-grv5.md` §1) un-gated the completion traces with `sema_dbg=Y` and +moved the second face from the wake-up to the counter publication (§6.1). ### 2.3 The mode matrix @@ -304,9 +306,10 @@ variants of the GR context commit, the launch method stream, the memory windows and the completion relay. The matrix exists because the rules the front-end and engine enforce are still being recovered; each knob isolates one variable so a supervised cycle can falsify one hypothesis at a time. Every non-neutral knob -is documented in `kernel/gsp_phys_gpu.c` (and, for the work-forwarding waves, -`main.c`) with its rationale and rollback; the parameter is the experiment's -record. +is documented in `kernel/gsp_phys_gpu.c` with its rationale and rollback +(the execution gates registered in `main.c`, e.g. `gsp_phys_gr_forward` and +the `sema_*` cadence knobs, follow the same rule); the parameter is the +experiment's record. ```mermaid flowchart LR @@ -440,14 +443,15 @@ The history matters because it explains the current code shape: are in the tree as commits `ebcfc1c2` (both-aperture kernel-native walk + content fallback), `7abd8f87` (FB-aperture discovery + per-channel re-arm, FORENSICS-grv1 R1) and `66388560` (periodic re-arm + FB-first fast - path, FORENSICS-grv3 R1) — the current HEAD. + path, FORENSICS-grv3 R1). In the current code (`svc_gr_content_discover`, `gsp_sema_service.c:7650`): -the FB scan runs first at a page stride (milliseconds), push candidates -persist across passes so a late ring can pair without re-walking 6 GiB, a -registration re-arms the epoch immediately, and hitting the pass cap starts a -30-second cooldown and then a fresh epoch while any registered rank still -lacks a ring — never the old permanent retirement. +the FB scan runs first and fast (the aperture is host RAM, so a page-stride +pass is milliseconds), passes are paced at one per 2 s (`:7731`), push +candidates persist across passes so a late ring can pair without re-walking +6 GiB, a registration re-arms the epoch immediately, and hitting the 60-pass +cap starts a 30-second cooldown and then a fresh epoch while any registered +rank still lacks a ring — never the old permanent retirement (`:7712-7734`). **Known limits.** The content path is gated to ranks 0–7 (`SVC_GR_CONTENT_RANKS=8`) and to the documented VA layout @@ -461,7 +465,7 @@ durable direction (ISSUE-0001 "durable direction", `issue-number-1.md:107`). ### 3.3 The semaphore service: release application `gsp_sema_service.c` is the vEP's engine room. Its service thread -(`svc_thread`, `:13287`) loops over the open mdev, runs the pump, retries +(`svc_thread`, `:13343`) loops over the open mdev, runs the pump, retries pending releases, and paces discovery. The release path: 1. **Decode** (`svc_decode_stream`, `:5170`) turns a consumed GPFIFO entry's @@ -471,7 +475,7 @@ pending releases, and paces discovery. The release path: resolver `svc_pool_pa_resolve` (`:2284`) — the resolver ISSUE-0001 fix #1 completed by adding the driver-published VAS walk (`svc_walk_vaspaces`, `:1274`); a miss is queued in `svc_pending_add` - (`:2472`) and retried every pass by `svc_apply_pending` (`:6201`), with the + (`:6166`) and retried every pass by `svc_apply_pending` (`:6201`), with the per-entry `tries` telemetry added by the same fix (`7f21daa7`). 3. **Apply** (`svc_sem_write`, `:2474`) reads the current word, writes only forward (never backwards), and is the sole software author of a release; @@ -481,27 +485,38 @@ pending releases, and paces discovery. The release path: The complementary **counter** path handles GR completion *counters* (distinct from CE tracking semaphores): `svc_counter_derive` (`:4658`) derives the -counter slot from structural translations; `svc_counter_autoreg` registers it -per GR channel; `svc_counter_release_apply` (`:4433`) applies decoded releases +counter slot from structural translations; `svc_counter_autoreg` (`:4710`) +registers it per GR channel by the fixed-total formula rank r → +`0x20400fec0 + (SVC_GR_COUNTER_TOTAL-1-r)*0x10` (`:4721`); +`svc_counter_release_apply` (`:4433`) applies decoded releases to the slot; `svc_counter_flush_pending` (`:4575`) lands payloads recorded before the slot was known; `svc_counter_publish` (`:9077`) writes the GPFIFO -PUT into the slot when it is ahead of the current value. Two facts learned the -hard way are now encoded: the counter pool total is fixed at 20 +PUT into the slot when it is ahead of the current value. Three facts learned +the hard way are now encoded: the counter pool total is fixed at 20 (`SVC_GR_COUNTER_TOTAL`, commit `fcb96912`) because a partial live channel -count poisons every later rank's slot; and the early publish is guarded so a -guessed PUT is never written (the H3 finding, `gsp_sema_service.c:11448-11477`, -FORENSICS-v2 §b/H3). +count poisons every later rank's slot; the early publish is guarded so a +guessed PUT is never written (the H3 finding, `gsp_sema_service.c:11489-11512`, +FORENSICS-v2 §b/H3); and the fixed-total mapping points each rank's slot at +the same word the ctx-init push releases to, so the publish becomes a +structural no-op (`put == counter_last`, `val <= cur`; `:9096-9109`) — the +grv5 stall ran with `counter publish` ×0 while `pub_calls=180142` and the +release, notifier clear and event post all fired (FORENSICS-grv5 §1, +verdict). The E1/E2 debug mirrors and GP_GET read-back for that finding +landed debug-gated as commit `c39b2a1c`. ### 3.4 Notifiers, GPFIFO consume, event delivery * **GPFIFO consume.** The pump reads `GP_PUT` from the USERD, walks entries - from its own GET, decodes, and advances; a replay-suppression path keeps + from its own GET, decodes, and advances (the GP_GET write-back is + `gsp_sema_service.c:12682-12684`, the end-of-pass publish `:12731-12732`); + a replay-suppression path keeps pin batches from re-consuming old slots (`gpfifo_skipped`), and a per-entry grace window holds an untranslatable GR entry before it is consumed as stale (`sema_gr_grace_ms`, default 8 s). `svc_userd_probe` (`:7099`) distinguishes "unreadable" from "readable but idle (0/0)" so a zero page is never mistaken for a live channel. -* **Notifiers.** `svc_notify_complete` (`:7207`) services the channel's error +* **Notifiers.** `svc_notify_complete` (`:7207`; the clear and event post are + `:7240-7277`) services the channel's error notifier token and posts the channel event. The notifier page address and aperture come from the channel allocation descriptor (`sema_channel_desc.notifier_pa/as`, `gsp_sema_service.h:31-32`), recorded @@ -512,22 +527,24 @@ FORENSICS-v2 §b/H3). the reply path injects MSI-X through the live eventfd (`gsp_emu_inject_completion_interrupt`). A green boot shows the sequence `notifier cleared … status=0xffff` (per GR channel) followed by the event - post; grv3 shows neither, and its three completed channels re-queued the - identical push and gave up — the leading reading is "counter moved, wake-up - never arrived" (FORENSICS-grv3 §3, hypothesis (d); not yet proven live - because the relevant traces are `sema_dbg`-gated). + post. grv3 (traces off) showed neither and re-queued the identical push; + the grv5 repeat with `sema_dbg=Y` settled the question — the notifier + cleared ×34 and the event posted, while the counter publish never wrote. + The leading reading is therefore the counter publication, not a missing + wake-up (FORENSICS-grv5 §1; §6.1). ### 3.5 Physical submit Forwarding is a two-stage ladder. The sema service captures a candidate and queues one request on the single ordered workqueue `gsp-phys-work` -(`svc_gr_request`, `:10991`; `svc_fwd_hold` is the single in-flight slot). +(`svc_gr_request`, `:10998`; `svc_fwd_hold` is the single in-flight slot). The work item calls into the backend: `gsp_phys_backend_guest_ce_copy` for CE (`gsp_phys_booter.c:15621`) or `gsp_phys_backend_guest_gr_submit` (`:16457`) -for GR, both gated by `gsp_phys_backend_work_ready()/ -gsp_phys_backend_gr_ready()` (`:14865`), which require READY, a live claim, -an unload not committed, a client-memory ladder channel, a work token, every -ladder VA, `gsp_phys_guest_mem` and an active guest RAM backing file. +for GR, gated by `gsp_phys_backend_work_ready()`/`gsp_phys_backend_gr_ready()` +(`:14849`/`:14865`), which require READY, a live claim, an unload not +committed, a client-memory ladder channel, a work token, and every ladder VA; +the GR submit additionally re-checks `gsp_phys_guest_mem` and the active +guest RAM backing at `:16509-16510`. GR submission (`gsp_phys_backend_guest_gr_submit`) is the vEP's most parameterized function: it relocates and pre-writes the QMD (inline-QMD @@ -558,10 +575,11 @@ forwarding-service cost, not engine time — the number to drive down is the | Both-aperture kernel-native GR discovery + content fallback | commit `ebcfc1c2`; `gsp_sema_service.c:7334-7529` | | FB-aperture discovery + re-arm; periodic re-arm + FB-first | commits `7abd8f87`, `66388560`; `gsp_sema_service.c:7650-7800` | | Ring not in guest RAM; RAM-only scan limitation | `FORENSICS-grv1.md` §0/§1/§A (local evidence) | -| 60-pass cap dead-end; wake-up hypothesis | `FORENSICS-grv3.md` §0/§3/§4/§5 (local evidence) | +| 60-pass cap dead-end; completion-chain questions | `FORENSICS-grv3.md` §0/§3/§4/§5; wake-up refuted and counter publication leading in `FORENSICS-grv5.md` §1/§4 (local evidence) | | CE clean consumption at 89 releases; no starvation | `FORENSICS-v2.md` §b/§c (local evidence) | | Pool resolver + starvation telemetry | commit `7f21daa7`; ISSUE-0001 fix #1/#2 | -| Counter slot fixed-total 20 | commit `fcb96912` | +| Counter slot fixed-total 20 | commit `fcb96912`; the fixed-total publish no-op is `FORENSICS-grv5.md` §1/§4 (local evidence) | +| Release written + notifier cleared + event posted while `counter publish` ×0 | `FORENSICS-grv5.md` §1/§4 (local evidence); commit `c39b2a1c` (E1/E2 debug mirrors) | | PROMOTE_CTX `bInitialize` must stay set | commit `88d47c1f`; ISSUE-0001 fix #3 | | GR exec latch + readiness gate | `gsp_phys_booter.c:14614`, `:14865`; `main.c:2609` | | PTE fetch-line rule, scratch pools | docs 11 §P6; `gsp_phys_booter.c:1044-1070` | @@ -626,7 +644,7 @@ are **ours**, written against the interfaces we implement. release; the poll observes them; the deferred release is applied once (P6b wave 1.5 semantics); the counter is published; the notifier is cleared; the channel event is posted; MSI-X wakes the guest. For GR the - last step is the open wake-up question (§6.1). + last step is the open counter-publication question (§6.1). --- @@ -663,8 +681,9 @@ kernels (mail, vadd) with correct release and clean teardown, on a lucky green boot; CE-class copies forwarded end-to-end with engine-authored release and fault relay (default off); compiled kernels of the P2 complexity are not yet observed executing through the stack. Pending: reproducibility -(ISSUE-0001), the GR descriptor-install wall (§6.1), completion wake-up -(§6.1), multiple kernels/clients per VM (ISSUE-0002), and P2/P3 in a green +(ISSUE-0001), the GR descriptor-install wall (§6.1), the GR counter-slot +publication / GP_GET write-back (§6.1; FORENSICS-grv5), multiple +kernels/clients per VM (ISSUE-0002), and P2/P3 in a green boot. The gating order is deliberate: reproducibility first, then the wall, then breadth, then performance. @@ -688,14 +707,20 @@ then breadth, then performance. nonzero `qmdvirt` hypothesis; m72 found the SKED `LOCAL_MEMORY_TOTAL_SIZE` cancel and added the `lmem_en` block; m91 showed the SM I-fetch MMU_NACK at shadow VAs and added `ptr_guest`). -* **The second face — completion wake-up.** FORENSICS-grv3 shows a boot where - discovery succeeded (three GR rings pinned), the ctx-init releases landed - (`gr_rel=3`, counter words = 1), and the guest *still* stopped: it - re-queued the identical push and gave up. The best-supported reading is - that the client waits for the completion *wake-up* (notifier clear + event), - not the counter value; green boots show `notifier cleared` ×N and the event - post, grv3 shows neither (FORENSICS-grv3 §3). The one-variable experiment is - a `sema_dbg=Y` cycle to un-gate the notifier/publish/post_event traces. +* **The second face — the completion-counter publication.** grv3 showed a + boot where discovery succeeded (three GR rings pinned), the ctx-init + releases landed (`gr_rel=3`, counter words = 1), and the guest *still* + stopped, re-queueing the identical push (FORENSICS-grv3 §0/§3). The grv5 + repeat ran with `sema_dbg=Y` and settled it: the release was written to the + exact pin-backed slot, `notifier cleared` fired ×34 and the channel event + posted (`posted=4`) — and `counter publish` still never wrote while + `pub_calls=180142`, because the fixed-total autoreg mapping makes the + publish a structural no-op (§3.3; FORENSICS-grv5 §1, verdict). The leading + reading is the counter-slot publication, with the GP_GET write-back as the + secondary, still-open candidate (the host writes `last_get` at + `gsp_sema_service.c:12682-12684` but nothing has read it back; + FORENSICS-grv5 §4/H2). Both observability surfaces are debug-gated today + (commit `c39b2a1c`). * **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The first fix wave landed (kernel-native pool translation `7f21daa7`, PROMOTE_CTX reply `88d47c1f`, issue register `a1407659`), and the @@ -729,8 +754,9 @@ then breadth, then performance. (`7abd8f87`, `66388560`). * **The notifier gate as a discovery symptom.** The `notifier cleared` line historically correlated with ring resolution; it is a publication-side - signal and is now understood as the wake-up face of §6.1, not a discovery - fix in itself. + signal and is now understood as part of the completion-publication face of + §6.1 (grv5: the notifier/event wake-up fires; the counter write is what is + missing), not a discovery fix in itself. ### 6.3 What is working well, and what is not @@ -742,15 +768,16 @@ mode-matrix discipline (one variable per supervised cycle, one-line rollback); the bounded-budget culture that turns potential freezes into logged, counted degradations. -**Not working:** GR dispatch (no grid observed); completion wake-up for GR; -reproducibility at the required 10/10 bar; second-client state isolation; -and the observability gaps that made both bugs slow to find — counter -`pub_calls` counting calls rather than writes (H3; the increment still sits -before the no-op guards at `gsp_sema_service.c:9085` and correction is -planned, FORENSICS-grv1 §5 R3), -`sema_dbg`-gated traces hiding the wake-up path, and the absence of a GR -error-reporting object on the forwarded path ("silence is not health", -doc 03-execution-frontier §8). +**Not working:** GR dispatch (no grid observed); the GR counter-slot +publication (structurally dead under the fixed-total mapping; FORENSICS-grv5 +§1); reproducibility at the required 10/10 bar; second-client state +isolation; and the observability gaps that made both bugs slow to find — +counter `pub_calls` counting calls rather than writes (H3; the increment +still sits before the no-op guards at `gsp_sema_service.c:9085` and correction +is planned, FORENSICS-grv1 §5 R3), completion traces still behind `sema_dbg` +(grv5 captured them, but the telemetry must become unconditional), and the +absence of a GR error-reporting object on the forwarded path ("silence is not +health", doc 03-execution-frontier §8). --- @@ -772,9 +799,10 @@ Every change follows the same evidence loop. This is the operating system of the vEP program, not an appendix: 1. **Question as a discriminating sentence + named acceptance signal.** E.g. - "does the notifier clear and the event post fire for a GR ctx-init - channel?" with the signal `notifier cleared` + `post_event` + guest - advancement past the re-queue. + "does the counter slot get published for a GR ctx-init channel?" with the + signal `counter publish … written` + guest advancement past the re-queue + (the earlier notifier/event question was answered by grv5: they fire; + FORENSICS-grv5 §1). 2. **Offline first.** Use the risc-v-gsp-rm-debugger (`debug/risc-v-gsp-rm-vm/`) and its validation tools — `harness/valprobe.py` for one firmware validation branch, `harness/rehost.py` for one whole function, RPC @@ -798,7 +826,7 @@ the vEP program, not an appendix: | GR rule recovery without burning cycles | risc-v-gsp-rm-debugger valprobe/rehost, RPC capture/replay (docs 45–47) | probe candidate front-end/engine rules offline; queue only falsifiable physical experiments | | Arbitrary-kernel acceptance | `tools/cuda-ab/p2/` (11 programs) | run in the first green boot after ISSUE-0001 acceptance; add a multi-process/multi-stream variant for ISSUE-0002 | | Performance measurement | `tools/cuda-ab/perf/` + `collect.py` | freeze thresholds on first green run; make the serialized `PDE_UPDATE` forwards the first optimization candidate | -| Wake-up observability | unconditional or counted `notified/posted` telemetry in `svc_notify_complete`/`gsp_rm_post_channel_event` (FORENSICS-grv3 §5B) | land as telemetry, not a behavior change; treat `posted==0` after a real clear as a first-class failure | +| Completion observability | unconditional or counted `notified`/`posted`/`publish-written` telemetry in `svc_notify_complete`, `gsp_rm_post_channel_event` and `svc_counter_publish` (FORENSICS-grv3 §5B, FORENSICS-grv5 §5) | land as telemetry, not a behavior change; treat `posted==0` after a real clear, or `publish written==0` with a registered slot, as a first-class failure; the E1/E2 debug mirrors (`c39b2a1c`) are the interim form | | Counter telemetry honesty | move the `pub_calls` increment after the write (FORENSICS-grv1 §5 R3) | land with the telemetry wave | | Multi-client state | ISSUE-0002 fix plan (per-client reset, per-mdev sema context, P0 hunks) | stage A1–A6 per the recorded plan; graduation criteria sequential/concurrent/simultaneous | | Evidence discipline | one evidence bundle per claim; counters as first-class discriminators; "no claim without a pair" | already the campaign norm (FORENSICS reports); keep counters stable enough to be quotable (see counter telemetry fix) | @@ -923,7 +951,8 @@ proactive design direction so the architecture is not surprised later. 5. **Observability as a product property.** The vEP's bugs were found by counter archaeology; counters that count the wrong thing (H3) cost days. Proactive design: every counter must answer one named question, and - telemetry for the wake-up path lands before it is needed again. + telemetry for the completion path (notifier, event, publish) lands before + it is needed again. 6. **Version drift under an adversarial schedule.** New images will arrive without notice; the triage/porting tooling exists but is unexercised on a real new version. Proactive design: golden sessions per supported version @@ -1026,4 +1055,5 @@ technical barrier between the project's architecture and its promise. | Date | Change | |---|---| -| 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. |
\ No newline at end of file +| 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. | +| 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (`svc_thread`, `svc_gr_request`, `svc_pending_add`, H3 guard, readiness gate); GR completion §3.4/§6.1 reconciled with FORENSICS-grv5 (wake-up fires; the counter publication is the leading reading); discovery cadence wording corrected (2-second paced passes, not "milliseconds"); mode-matrix source note corrected. | |