debug: green-eye wave 3 - RM harness, local simulator, device map, MCP plugin, value-validation demos, CI pipeline, toolchain docs; add host RM tools rmtool/rmtop

6 files changed, 402 insertions, 0 deletions

diff --git a/debug/risc-v-gsp-rm-vm/rmharness/README.md b/debug/risc-v-gsp-rm-vm/rmharness/README.md
new file mode 100644
index 0000000..6d33138
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/rmharness/README.md
@@ -0,0 +1,117 @@
+# rmharness - host RM oracle integration for green-eye
+
+`rmharness` wires the Open-IOV host RM userspace tools (`rmtool`, `rmtop`,
+`rmq`) into the green-eye workflow so host ground truth can be captured next
+to simulator sessions and compared later (docs/06-ACCURACY.md M1/M6).
+
+All three tools are **read-only**: they query the host RM through
+`/dev/nvidiactl` control ioctls plus `/proc` and `/sys` reads. The harness
+never loads/unloads modules, never binds/unbinds the driver, never creates
+mdevs, never claims the GPU and never touches `/sys/fs/pstore`.
+
+## Tools
+
+| wrapper | tool | what it is for |
+|---|---|---|
+| `rmtool.sh` | `tools/rmtool` | CLI oracle: GPU/engine inventory, VRAM/heap/BAR1 stats, allowlisted NV2080 controls, GPU timer sampling |
+| `rmtop.sh` | `tools/rmtop` | ncurses dashboard; `--once` renders the same panels as table/json/csv for capture |
+| `rmq.sh` | `tools/rmq` | the original host RM oracle (device info tables); unmodified |
+
+## Layout
+
+```
+rmharness/
+  README.md             this file
+  rmtool.sh rmtop.sh rmq.sh
+                        build-if-needed wrappers (pass arguments through)
+  harness.sh            run a wrapper, capture stdout/stderr, record a session
+  tests/test_rmharness_plugin.py
+                        fake-`ge` plugin import/registration check
+```
+
+The green-eye plugin lives next to the other bundled plugins:
+`plugins/rmharness_plugin.py` (REPL commands `rmtool`, `rmtop`, `rmq`).
+
+## Build and smoke
+
+The wrappers build their tool on first use. To build explicitly:
+
+```bash
+make -C tools/rmtool
+make -C tools/rmtop
+make -C tools/rmq          # already existed; unchanged
+```
+
+Safe smoke checks (no RM ioctl, no device access):
+
+```bash
+./rmharness/rmtool.sh --help
+./rmharness/rmtool.sh --version
+./rmharness/rmtool.sh gpu list          # reads /proc/driver/nvidia only
+./rmharness/rmtool.sh gpu topology      # reads /sys/bus/pci only
+./rmharness/rmtop.sh --once             # read-only RM queries (see note)
+```
+
+`rmtop --once` and `gpu info`/`engine list`/`mem stats`/`sample timer` open a
+short-lived RM client (exactly what `nvidia-smi` does) and issue read-only
+controls. They do not disturb a running CUDA job. If the driver is not bound
+they fail cleanly. Commands that would allocate or mutate anything are refused
+by the tools themselves.
+
+## Capturing into a green-eye session
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+./rmharness/harness.sh --session dumps/my-run --name before \
+    -- rmtool engine list --format=json
+./rmharness/harness.sh --session dumps/my-run -- rmtop --once
+```
+
+`harness.sh` runs the wrapper, streams stdout/stderr through, writes the
+captured streams to `<session>/blobs/`, and appends one line to
+`<session>/events.jsonl`:
+
+```json
+{"type": "rm_harness_run", "tool": "rmtool", "argv": [...], "exit": 0,
+ "duration_s": 0.02, "stdout": {"path": "blobs/...", "size": 123,
+ "sha256": "..."}, "stderr": {...}, "readonly": true}
+```
+
+Recording is additive: existing sessions are never rewritten. If the
+directory has no `session.json`, a minimal `greeneye-session/1` one is
+created. The event `t` field is `0.0` for externally recorded runs; use
+`start`/`end` for wall-clock ordering.
+
+## Plugin
+
+`plugins/rmharness_plugin.py` registers REPL commands that run the wrappers
+and record the output through the live recorder:
+
+```
+greeneye> rmtool gpu list
+greeneye> rmtool mem stats --format=json
+greeneye> rmtop --once
+greeneye> rmq
+```
+
+Arguments naming write operations (`alloc`, `map`, `pin`, `start`, ...) are
+refused and recorded as `rm_harness_skip`. With a null recorder the plugin
+still runs and prints; it just does not persist.
+
+Check it without QEMU:
+
+```bash
+python3 rmharness/tests/test_rmharness_plugin.py
+```
+
+## Not implemented / not run
+
+* `rmq` device-info tables (`FIFO_GET_DEVICE_INFO_TABLE`) and
+  `nv2080 class list` return `NV_ERR_INSUFFICIENT_PERMISSIONS` /
+  `NV_ERR_NOT_SUPPORTED` for an unprivileged client on this driver; the tools
+  report the status instead of escalating.
+* `rmtool channel/context/resserve/cudasw/resman/gsp` and sampling of
+  clocks/power/thermal are documented but not implemented in the minimal
+  read-only build; they return a clear "not implemented / refused" status.
+* Nothing in this harness was validated against a live migration or a second
+  GPU; multi-GPU selection (`-d N`) uses RM `deviceId` = minor.
diff --git a/debug/risc-v-gsp-rm-vm/rmharness/harness.sh b/debug/risc-v-gsp-rm-vm/rmharness/harness.sh
new file mode 100755
index 0000000..60fce8a
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/rmharness/harness.sh
@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+# harness.sh - run a read-only RM tool, capture stdout/stderr, and record the
+# run into a green-eye session directory (blob + events.jsonl line).
+#
+# usage: harness.sh [--session DIR] [--name LABEL] -- TOOL [ARGS...]
+#        TOOL is one of: rmtool rmtop rmq
+#
+# When --session is given the harness appends a "rm_harness_run" event and
+# writes the captured streams under <session>/blobs/. Existing session files
+# are never rewritten; recording is additive. Without --session the tool runs
+# and streams pass through unchanged.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ALLOWED="rmtool rmtop rmq"
+
+usage() {
+	cat <<'EOF'
+usage: harness.sh [--session DIR] [--name LABEL] -- TOOL [ARGS...]
+
+TOOL is one of: rmtool rmtop rmq (wrapper scripts in this directory).
+Everything after -- is passed to the tool unchanged.
+EOF
+}
+
+session=""
+label=""
+while [[ $# -gt 0 ]]; do
+	case "$1" in
+	--session)
+		session="${2:?--session needs a directory}"
+		shift 2
+		;;
+	--name)
+		label="${2:?--name needs a label}"
+		shift 2
+		;;
+	-h | --help)
+		usage
+		exit 0
+		;;
+	--)
+		shift
+		break
+		;;
+	-*)
+		echo "harness.sh: unknown option '$1'" >&2
+		usage >&2
+		exit 2
+		;;
+	*)
+		break
+		;;
+	esac
+done
+
+tool="${1:-}"
+if [[ -z "$tool" ]]; then
+	usage >&2
+	exit 2
+fi
+shift
+tool="${tool%.sh}"
+case " $ALLOWED " in
+*" $tool "*) ;;
+*)
+	echo "harness.sh: tool '$tool' is not on the read-only allowlist" \
+		"($ALLOWED)" >&2
+	exit 2
+	;;
+esac
+
+wrapper="$HERE/$tool.sh"
+if [[ ! -x "$wrapper" ]]; then
+	echo "harness.sh: wrapper not found: $wrapper" >&2
+	exit 2
+fi
+
+argv_json="$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1:]))' \
+	"$tool" "$@")"
+out_file="$(mktemp)"
+err_file="$(mktemp)"
+trap 'rm -f "$out_file" "$err_file"' EXIT
+
+start="$(python3 -c 'import time; print(time.time())')"
+rc=0
+"$wrapper" "$@" >"$out_file" 2>"$err_file" || rc=$?
+end="$(python3 -c 'import time; print(time.time())')"
+
+cat "$out_file"
+cat "$err_file" >&2
+
+if [[ -n "$session" ]]; then
+	mkdir -p "$session/blobs"
+	stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+	out_rel="blobs/rmharness-${stamp}-${tool}-stdout.txt"
+	err_rel="blobs/rmharness-${stamp}-${tool}-stderr.txt"
+	cp "$out_file" "$session/$out_rel"
+	cp "$err_file" "$session/$err_rel"
+	SESSION="$session" TOOL="$tool" ARGV_JSON="$argv_json" \
+		RC="$rc" START="$start" END="$end" LABEL="$label" \
+		OUT_REL="$out_rel" ERR_REL="$err_rel" \
+		python3 - <<'PY'
+import hashlib
+import json
+import os
+import time
+
+session = os.environ["SESSION"]
+
+
+def blob_info(rel):
+    path = os.path.join(session, rel)
+    with open(path, "rb") as fh:
+        data = fh.read()
+    return {"path": rel, "size": len(data),
+            "sha256": hashlib.sha256(data).hexdigest()}
+
+
+event = {
+    "t": 0.0,
+    "type": "rm_harness_run",
+    "tool": os.environ["TOOL"],
+    "label": os.environ["LABEL"] or None,
+    "argv": json.loads(os.environ["ARGV_JSON"]),
+    "exit": int(os.environ["RC"]),
+    "start": float(os.environ["START"]),
+    "end": float(os.environ["END"]),
+    "duration_s": round(float(os.environ["END"]) - float(os.environ["START"]), 6),
+    "stdout": blob_info(os.environ["OUT_REL"]),
+    "stderr": blob_info(os.environ["ERR_REL"]),
+    "readonly": True,
+}
+meta = os.path.join(session, "session.json")
+if not os.path.exists(meta):
+    with open(meta, "w", encoding="utf-8") as fh:
+        json.dump({"format": "greeneye-session/1", "name": "rmharness",
+                   "created": time.time(),
+                   "greeneye": "external-rmharness/1"}, fh, indent=1,
+                  sort_keys=True)
+with open(os.path.join(session, "events.jsonl"), "a", encoding="utf-8") as fh:
+    fh.write(json.dumps(event, sort_keys=True) + "\n")
+print(f"[rmharness] recorded {os.environ['TOOL']} -> {session} "
+      f"({os.environ['OUT_REL']})")
+PY
+fi
+
+exit "$rc"
diff --git a/debug/risc-v-gsp-rm-vm/rmharness/rmq.sh b/debug/risc-v-gsp-rm-vm/rmharness/rmq.sh
new file mode 100755
index 0000000..ce0670e
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/rmharness/rmq.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# rmq.sh - build-if-needed wrapper for the existing tools/rmq host RM oracle.
+# rmq itself is not modified by the harness; this only locates/builds it.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "$HERE/../../.." && pwd)"
+BIN="$ROOT/tools/rmq/rmq"
+# rmq has no --help flag; answer it here so the smoke check never touches the
+# device (see rmharness/README.md "Safe smoke checks").
+case "${1:-}" in
+-h | --help)
+	echo "usage: $(basename "$0") [rmq-index options]   (read-only RM oracle)"
+	echo "  rmq has no launcher of its own; see $ROOT/tools/rmq/."
+	echo "  Built on first use from $ROOT/tools/rmq."
+	exit 0
+	;;
+esac
+if [[ ! -x "$BIN" ]]; then
+	make -C "$ROOT/tools/rmq" >&2
+fi
+exec "$BIN" "$@"
diff --git a/debug/risc-v-gsp-rm-vm/rmharness/rmtool.sh b/debug/risc-v-gsp-rm-vm/rmharness/rmtool.sh
new file mode 100755
index 0000000..405a766
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/rmharness/rmtool.sh
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# rmtool.sh - build-if-needed wrapper for the Open-IOV rmtool RM oracle.
+# Read-only tool; see ../docs/17-RM-HARNESS.md for the safety contract.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "$HERE/../../.." && pwd)"
+BIN="$ROOT/tools/rmtool/rmtool"
+if [[ ! -x "$BIN" ]]; then
+	make -C "$ROOT/tools/rmtool" >&2
+fi
+exec "$BIN" "$@"
diff --git a/debug/risc-v-gsp-rm-vm/rmharness/rmtop.sh b/debug/risc-v-gsp-rm-vm/rmharness/rmtop.sh
new file mode 100755
index 0000000..5cde6c7
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/rmharness/rmtop.sh
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# rmtop.sh - build-if-needed wrapper for the Open-IOV rmtop dashboard.
+# Use --once for non-interactive capture; interactive mode needs a TTY.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "$HERE/../../.." && pwd)"
+BIN="$ROOT/tools/rmtop/rmtop"
+if [[ ! -x "$BIN" ]]; then
+	make -C "$ROOT/tools/rmtop" >&2
+fi
+exec "$BIN" "$@"
diff --git a/debug/risc-v-gsp-rm-vm/rmharness/tests/test_rmharness_plugin.py b/debug/risc-v-gsp-rm-vm/rmharness/tests/test_rmharness_plugin.py
new file mode 100644
index 0000000..8d74f6e
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/rmharness/tests/test_rmharness_plugin.py
@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Unit-style check for plugins/rmharness_plugin.py (stdlib only).
+
+Run from the risc-v-gsp-rm-vm root:
+
+    python3 rmharness/tests/test_rmharness_plugin.py
+
+It loads the plugin with a fake ``ge`` object and verifies:
+  * register() adds the rmtool/rmtop/rmq commands;
+  * write arguments are refused and recorded as rm_harness_skip;
+  * a real ``rmtool --version`` run is recorded as rm_harness_run with a blob.
+"""
+from __future__ import annotations
+
+import importlib.util
+import os
+import unittest
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_TOOL_ROOT = os.path.dirname(os.path.dirname(_HERE))
+_PLUGIN = os.path.join(_TOOL_ROOT, "plugins", "rmharness_plugin.py")
+
+
+class FakeRecorder:
+    def __init__(self, directory=None):
+        self.dir = directory
+        self.events = []
+        self.blobs = []
+
+    def event(self, event_type, **fields):
+        self.events.append(dict(type=event_type, **fields))
+
+    def blob(self, name, data):
+        self.blobs.append((name, data))
+        return name
+
+
+class FakeGe:
+    def __init__(self, directory=None):
+        self.recorder = FakeRecorder(directory)
+        self.commands = {}
+        self.logs = []
+
+    def log(self, message, **fields):
+        self.logs.append(message)
+
+    def add_command(self, name, fn, help=""):
+        self.commands[name] = (fn, help)
+
+
+def load_plugin():
+    spec = importlib.util.spec_from_file_location("rmharness_plugin_test",
+                                                  _PLUGIN)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+class RmHarnessPluginTest(unittest.TestCase):
+    def setUp(self):
+        self.plugin = load_plugin()
+        self.ge = FakeGe()
+        self.plugin.register(self.ge)
+
+    def test_registers_commands(self):
+        self.assertEqual(sorted(self.ge.commands),
+                         ["rmq", "rmtool", "rmtop"])
+        self.assertTrue(any("rmharness" in msg for msg in self.ge.logs))
+
+    def test_blocked_write_is_refused(self):
+        fn, _help = self.ge.commands["rmtool"]
+        rc = fn(None, ["mem", "alloc", "-d", "0", "--size=1G"])
+        self.assertEqual(rc, 2)
+        types = [e["type"] for e in self.ge.recorder.events]
+        self.assertIn("rm_harness_skip", types)
+        self.assertNotIn("rm_harness_run", types)
+
+    def test_readonly_run_is_recorded_with_blob(self):
+        if not os.path.isfile(
+                os.path.join(_TOOL_ROOT, "rmharness", "rmtool.sh")):
+            self.skipTest("rmharness wrapper not present")
+        fn, _help = self.ge.commands["rmtool"]
+        rc = fn(None, ["--version"])
+        self.assertEqual(rc, 0)
+        types = [e["type"] for e in self.ge.recorder.events]
+        self.assertIn("rm_harness_run", types)
+        self.assertTrue(self.ge.recorder.blobs)
+        run = [e for e in self.ge.recorder.events
+               if e["type"] == "rm_harness_run"][0]
+        self.assertTrue(run["readonly"])
+        self.assertEqual(run["tool"], "rmtool")
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=2)