From d4d3c7ded20edbb1d3fe37a500604aa7cd6b8e93 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 1 Oct 2026 11:52:17 -0400 Subject: STATE-CURRENT: cycles 2-3 results - clean release without reboot; RM init failure 0x59 under diagnosis --- STATE-CURRENT.md | 94 ++++++++++++++++++++++++++++++++------------------------ 1 file changed, 54 insertions(+), 40 deletions(-) diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 85fb904..156f35a 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-01 11:30.** This is a dated public status page, not live +> **Snapshot: 2026-10-01 12:00.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -64,9 +64,21 @@ chain succeeds. section below), and a second live cycle on the rung-4 build repeated the boot and reached the server's run-sequencer request; the executor's size cap rejected the real request value (16354), which was diagnosed from a - raw host-memory read of the message, fixed, and rebuilt; the claimed - session from that cycle is awaiting a supervised power cycle to run the - corrected boot+shutdown flow. + raw host-memory read of the message, fixed, and rebuilt. The post-reboot + recovery is now verified and the zero-timeout REG_POLL default fix + (commit `98b861d`, build `87309e3a…`) was confirmed live. Cycles 2-3 + (2026-10-01 11:40 and 11:46) then reached **stage 3e on real hardware**: + the real GSP-RM executed the full 165-command/490-dword sequencer and + reported RM init failure `0x59` (`NV_ERR_OPERATING_SYSTEM`) on both + firmware pairings (the default 570.211.01 image and a verified matching + 570.144 image - the image-pairing hypothesis is disproven). Each failed + cycle now **releases cleanly back to the distribution driver without a + reboot**: the server's own shutdown sets the remaining secure-scratch + handoff bit, so all four handoff bits are set and a single `rmmod` clears + the protected region, rebinds the distribution driver and releases the + power-domain locks. The next evidence is the **GSP-RM NOCAT journal** (the + matching-image cycle streamed ~851 journal records), whose capture is + being built. - **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload @@ -210,47 +222,49 @@ observed on the physical TU104: command buffer the host must execute; `GSP_INIT_DONE` (0x1001) follows only after that sequencer runs and was not observed in the 60 s poll. -This is the first host↔real-GSP-RM exchange on the physical path. A clean -release is not yet possible: the Booter Unload gate requires all four secure -scratch14 handoff bits (24–27); a host-driven boot sets only bit 24, and bit -27 is set by the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`, -not yet implemented). The host-side seed/unlock alternative is fuse-locked on -this part. The session is therefore intentionally left claimed (`--keep`): the -GPU is bound to `open-iov-gsp-phys` and runtime-active; `nvidia` stays unbound -(host headless; `nvidia-smi` fails by design); host healthy, pstore empty. A -fresh live cycle needs a power cycle. +This was the first host↔real-GSP-RM exchange on the physical path. At that +point a clean release was not yet possible: the Booter Unload gate requires +all four secure scratch14 handoff bits (24–27); a host-driven boot set only +bit 24, and bit 27 was expected from the server's shutdown path (host RPC +`UNLOADING_GUEST_DRIVER`, not yet implemented then). The host-side seed/unlock +alternative is fuse-locked on this part. The session was therefore +intentionally left claimed (`--keep`): the GPU was bound to +`open-iov-gsp-phys` and runtime-active; `nvidia` stayed unbound (host +headless; `nvidia-smi` fails by design); host healthy, pstore empty. A fresh +live cycle needed a power cycle at that point. **Status update (cycles 2-3, +same day):** this limitation is retired - the server's shutdown set the +remaining handoff bit and a single `rmmod` released the device without a +reboot (see the frontier section). **Next steps:** (a) execute the 0x1002 sequencer command buffer in the module and log 0x1001 as `stage 3e complete`; (b) send `UNLOADING_GUEST_DRIVER` on the command queue so the server shuts down, sets the remaining handoff bit and teardown can hand the GPU back without a reboot; (c) prove a request/reply -round trip with the server. All three are now implemented but not yet loaded: -the module sources contain the sequencer executor (BAR0 register-write/modify/ -poll, delay, store and core-resume opcodes), a generalized msgq producer -(multi-element records, checksum/sequence/free-space rules, doorbell) with the -unload RPC, a reply-matching status-queue reader, and a debugfs probe that -requests the server's static configuration after init-done. The rebuilt module -awaits the next supervised live cycle (which requires a power cycle, with -approval, because the current claimed session cannot be released); the rung -4a/4b/4c code passed an adversarial wire-format review plus its fixes. - -**Prepared for the next cycle (build, not loaded):** `open-iov-vfio.ko` sha256 -`f61308bd…`; the cycle should run `phys_claim_cycle.sh --yes --hold 60 ---cycles 1 --boot --keep`, then exercise `gsp-phys-rm-probe` (static-info -round trip) and `gsp-phys-unload-rpc` (server shutdown) through the module's -read-only/debug state files before a clean `rmmod`. That cycle has now run: -the boot repeated green, and the server's run-sequencer request was received -but rejected by an over-strict host-side cap on the request's own buffer size. -The message was read raw from host memory, the cap corrected and the module -rebuilt; because the claimed session cannot be released without the shutdown -RPC, the next supervised power cycle will run the corrected boot-and-shutdown -flow. - -Ending the claimed session (harness notice): `sudo rmmod open_iov_vfio`, then -`echo auto | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the -same for `0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. -Because the shutdown RPC is missing, a plain release cannot pass the handoff -gate, so the reliable route back to the distribution driver is a power cycle. +round trip with the server. All three were implemented, built and have now +run live: cycles 2-3 executed the sequencer and reached init-done, and the +server's own shutdown supplied the missing handoff bit, so each failed cycle +released without a reboot; a request/reply round trip remains to be +exercised. The module sources contain the sequencer executor (BAR0 +register-write/modify/poll, delay, store and core-resume opcodes), a +generalized msgq producer (multi-element records, +checksum/sequence/free-space rules, doorbell) with the unload RPC, a +reply-matching status-queue reader, and a debugfs probe that requests the +server's static configuration after init-done; the rung 4a/4b/4c code passed +an adversarial wire-format review plus its fixes. + +**Prepared for the next cycle (now history):** an early build (`f61308bd…`) +ran the boot green but the server's run-sequencer request was rejected by an +over-strict host-side cap on the request's own buffer size. The message was +read raw from host memory, the cap corrected (build `87309e3a…`), and the +sequencer then ran live in cycles 2-3 (see the frontier section). + +Ending a claimed session: `sudo rmmod open_iov_vfio`, then `echo auto | sudo +tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the same for +`0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. Since cycles +2-3 this passes the handoff gate (the server's shutdown sets the last handoff +bit) and completes without a power cycle; if the protected region survives +teardown the fail-safe keeps the locks and a power cycle is still required, +with approval. ## Initialization, work and teardown path -- cgit v1.2.3