From 002e3d7d13c414ea874abbee8d82803dd3a72fe5 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 10:58:21 -0400 Subject: debug: green-eye wave 3 - RM harness, local simulator, device map, MCP plugin, value-validation demos, CI pipeline, toolchain docs; add host RM tools rmtool/rmtop --- debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md | 164 +++++++++++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md (limited to 'debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md') diff --git a/debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md b/debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md new file mode 100644 index 0000000..59db437 --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md @@ -0,0 +1,164 @@ +# 17 — RM host oracle harness (`rmharness`) + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status. +> The RM tools are read-only host diagnostics. They are development tooling: +> never shipped, never present in a guest, never a driver dependency. + +## 1. Purpose + +The GSP emulation can only be trusted against ground truth. Two of the +metrics in [06-ACCURACY](06-ACCURACY.md) need the real host RM as a reference: + +* **M1 — inventory coverage**: which controls/engines the real RM exposes; +* **M6 — memory side-effects / fixed points**: real VRAM/heap/BAR1 numbers + and engine inventory at a moment in time. + +`rmharness` packages the host-side oracle tools and wires their output into +green-eye sessions, so a simulator/emulation session and a host RM capture +can be recorded, diffed and cited together. It is the host half of the +accuracy loop (H4.5), not a boot path or a replacement for PLAN B traces. + +## 2. Tool inventory + +| tool | source | role in the loop | +|---|---|---| +| `rmtool` | `tools/rmtool/` (new) | CLI oracle: `gpu list/topology/info`, `engine list/info`, `mem stats`, `nv2080 class list`, allowlisted `nv2080 method call`, `sample timer`; `--format=table\|json\|csv` | +| `rmtop` | `tools/rmtop/` (new) | ncurses dashboard over the same query layer; `--once` renders table/json/csv for capture | +| `rmq` | `tools/rmq/` (pre-existing, unmodified) | FIFO/INTERNAL device info tables for TU104 static data | + +Both new tools share one ioctl layer (`tools/rmtool/src/rm_api.c`, +`query.c`): version check, `NV01_ROOT_CLIENT` -> `NV01_DEVICE_0` -> +`NV20_SUBDEVICE_0` alloc ladder, and `NV_ESC_RM_CONTROL` queries. The +device-open protocol follows what `nvidia-smi` does: open +`/dev/nvidia`, `NV_ESC_WAIT_OPEN_COMPLETE`, then allocate on +`/dev/nvidiactl`. The ABI structs match the running driver (the in-tree +userspace SDK snapshot is older and its 40-byte `NVOS64` is rejected). + +## 3. Build + +```bash +make -C tools/rmtool # builds rmtool + lib/librmapi.a +make -C tools/rmtop # builds rmtop (ncursesw) +make -C tools/rmq # unchanged, still builds +``` + +Requirements: gcc, ncursesw for rmtop. No external SDK, NVML or +libnvidia-ml linkage. `make clean` removes all artifacts. + +## 4. Running + +```bash +tools/rmtool/rmtool --help +tools/rmtool/rmtool gpu list --format=json +tools/rmtool/rmtool gpu info -d 0 +tools/rmtool/rmtool engine list +tools/rmtool/rmtool mem stats +tools/rmtool/rmtool sample timer --interval=100ms --count=20 +tools/rmtop/rmtop --once +tools/rmtop/rmtop -g 0 --interval=250 # interactive (needs a TTY) +``` + +`gpu list`/`gpu topology` need no device access (they read +`/proc/driver/nvidia/gpus` and PCI sysfs). Everything else opens a +short-lived RM client and issues read-only controls; this is the same +access model as `nvidia-smi` and is safe alongside a running CUDA job. + +`--device-spec`/`-d` accepts a minor (`0`) or a bus id (`0000:01:00.0`) and +maps to the RM `deviceId`; `--device PATH` overrides the `/dev/nvidiaN` +node. + +## 5. green-eye integration + +Wrappers `rmharness/rmtool.sh`, `rmharness/rmtop.sh`, `rmharness/rmq.sh` +locate and build the tools and pass arguments through. `rmharness/harness.sh` +adds capture: + +```bash +cd debug/risc-v-gsp-rm-vm +./rmharness/harness.sh --session dumps/real-rm-before --name before \ + -- rmtool engine list --format=json +./rmharness/harness.sh --session dumps/real-rm-before -- rmtop --once +``` + +The harness appends one `rm_harness_run` event per run and stores +stdout/stderr under `/blobs/` with sizes and SHA-256. Existing +sessions are appended to, never rewritten; a missing `session.json` is +created with the `greeneye-session/1` format. Because these runs are +external, the event `t` field is `0.0`; `start`/`end` (epoch seconds) carry +the wall clock. `greeneye analyze` tolerates the session; the run count shows +up in the event summary. + +`plugins/rmharness_plugin.py` exposes the same wrappers in the REPL +(`rmtool`, `rmtop`, `rmq`), recording through `ge.recorder.blob()` + +`ge.recorder.event()`. Write-looking arguments are refused and recorded as +`rm_harness_skip`. The plugin is stdlib-only and is exercised without QEMU +by `rmharness/tests/test_rmharness_plugin.py` (fake `ge`). + +## 6. Safety contract (absolute) + +* Host-side read-only queries only. The tools allocate only an RM client / + device / subdevice handle pair; they never create engine, channel or memory + objects. +* Never load/unload kernel modules, bind/unbind the driver, create/destroy + mdevs, claim the physical GPU, reboot, or touch `/sys/fs/pstore`. +* Write paths are enforced in code, not by convention: + * `rmtool` refuses `mem alloc/map/unmap/pin`, `engine allocate/start/stop/ + config`, and `channel/context/resserve/cudasw/resman/gsp` with exit 4/3; + * `nv2080 method call` only sends controls on a read-only allowlist + (`GPU_GET_INFO_V2`, `GPU_GET_NAME_STRING`, `GPU_GET_ENGINES_V2`, + `MC_GET_ARCH_INFO`, `FB_GET_INFO_V2`, `BUS_GET_PCI_INFO`, + `TIMER_GET_TIME`, `FIFO_GET_DEVICE_INFO_TABLE`, + `GPU_GET_CLASSLIST_V2`, ...) and refuses anything else before the ioctl; + * `harness.sh` only accepts `rmtool`, `rmtop`, `rmq`; + * the plugin blocks write-looking arguments. +* Another agent may be running a live CUDA/PLAN B job; the tools do not + touch its context and all smoke checks above run without a GPU claim. + +## 7. What was and was not run + +Smoke checks executed on the development host (driver 570.211.01, TU104 +RTX 2080 Super Max-Q, one GPU at `0000:01:00.0`): + +* `--help`/`--version` for both tools; `gpu list`, `gpu topology` (proc/sysfs + only); +* `gpu info`, `engine list`, `mem stats`, `sample timer`, `rmtop --once` + (read-only RM controls) — all returned live data (`TU100`, 10 engines GR + + 5 CE + NVDEC + NVENC + SW + SEC2, 8 GiB VRAM, BAR1 256 MiB); +* `harness.sh` session recording and the plugin test. + +Not run, and why: `rmq` device info tables and `nv2080 class list` return +`NV_ERR_INSUFFICIENT_PERMISSIONS` / `NV_ERR_NOT_SUPPORTED` for an +unprivileged client on this driver (reported, not escalated); interactive +`rmtop` was not driven (no TTY in the build shell); nothing here exercises +migrations, MIG or a second GPU. + +## 8. Roadmap tie-ins + +* **docs/06-ACCURACY** — recorded `rm_harness_run` blobs are the host + reference for M1/M6; keep the JSON captures in the session directories + cited by the metric table. +* **GREEN-EYE-ROADMAP H3** — the custom simulator's device map and RM-core + interception need real control semantics (IDs, sizes, status values) to + validate against; `rmtool` supplies that inventory before the H2.2 boot is + reached, and the shared `rm_api.c` can become the simulator's host-RM + client for differential tests. +* **GREEN-EYE-ROADMAP H4.5** — the canonical observe -> compare -> implement + loop can now include a host capture (`harness.sh --session`) beside the + simulator session, giving every emulation change an evidence pair. +* **R4 (full-memory trace)** — the session-event schema already carries + external runs; when the TCG trace lands, host `rm_harness_run` events and + firmware `access` events can be interleaved for a complete real-vs-sim + picture. +* **H1.3** — injected services that emulate RM controls should be diffed + field-by-field against the `rmtool nv2080 method call` reply for the same + control id. + +## 9. Extending safely + +1. Add new read-only controls to the allowlist **and** to `rmtool.h` + structs, with the exact kernel-side size (the userspace SDK snapshot is + not authoritative). +2. Never add an override that bypasses the allowlist/refusals. +3. Record any new query in the session and document the skip reason for + anything privilege-gated. +4. Keep `tools/rmq/` untouched; call it from the harness. -- cgit v1.2.3