From 68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- docs/tools/rmtop.md | 227 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 docs/tools/rmtop.md (limited to 'docs/tools/rmtop.md') diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md new file mode 100644 index 0000000..6408268 --- /dev/null +++ b/docs/tools/rmtop.md @@ -0,0 +1,227 @@ +# rmtop - GPU htop Equivalent + +## Overview +`rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry. + +## Source Code Reuse +Same recycled components as `rmtool`: +- `gpumon_perfmon` - sampling infrastructure, perfmon counters +- `nvidia-smi` - GPU enumeration, NVML-style queries +- `nv2080` - direct RM control for real-time state +- `resserve` - resource server state +- `resman` - resource manager heaps/clients + +## UI Layout + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ rmtop 1.0 │ GPU 0: RTX 5090 (GB202) │ Driver: 610.XX │ 14:32:15 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ ENGINES │ UTIL │ FREQ │ POWER │ TEMP │ ACTIVE CHANNELS │ +│ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │ 320W │ 78°C │ ch: 0,3,7,12 │ +│ ├─ CE0 (Copy) │ ████░░░░░░ 41% │ 2100MHz │ │ │ ch: 1,4 │ +│ ├─ CE1 (Copy) │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVDEC0 │ ██████████ 100% │ 1800MHz │ │ │ ch: 2 │ +│ ├─ NVDEC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVENC0 │ ████████░░ 82% │ 1800MHz │ │ │ ch: 5 │ +│ ├─ NVENC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVJPG │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ └─ OFA │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ MEMORY │ USED │ TOTAL │ FREE │ FRAG │ BAR1 │ +│ ├─ VRAM │ 18.2G │ 24.0G │ 5.8G │ 12% │ 256M / 256M │ +│ ├─ System RAM │ 4.1G │ 64.0G │ 59.9G │ 3% │ │ +│ └─ BAR1 (CPU) │ 12M │ 256M │ 244M │ 1% │ │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ CHANNELS / CONTEXTS (PID │ ENG │ CTX │ VRAM │ SYSMEM │ UTIL │ STATE) │ +│ ├─ 1247 │ GR │ 0x3 │ 2.1G │ 512M │ 82% │ RUNNING │ +│ ├─ 1247 │ CE0 │ 0x3 │ 128M │ 64M │ 41% │ RUNNING │ +│ ├─ 1247 │ DEC │ 0x3 │ 4.0G │ 256M │ 100% │ RUNNING │ +│ ├─ 1247 │ ENC │ 0x3 │ 1.5G │ 128M │ 82% │ RUNNING │ +│ ├─ 3891 │ GR │ 0x7 │ 512M │ 128M │ 12% │ WAITING │ +│ └─ 3891 │ CE0 │ 0x7 │ 64M │ 32M │ 5% │ WAITING │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ POWER / CLOCKS / THERMALS │ +│ ├─ Power Draw: 320W / 450W (71%) │ GPU Clock: 2850 MHz (max) │ +│ ├─ Mem Clock: 1313 MHz (max) │ SM Clock: 2850 MHz │ +│ ├─ Temp: 78°C (slowdown: 83°C) │ Fan: 65% (auto) │ +│ └─ PState: P0 (Performance) │ Perf Cap: None │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ GSP-RM TELEMETRY │ +│ ├─ RPC Queue: GR=12 CE=3 DEC=0 ENC=1 (depth / max=64) │ +│ ├─ Doorbell Rate: 45K/s │ Firmware Threads: 8/16 active │ +│ ├─ WPR Regions: 3 active │ Migration: IDLE │ +│ └─ Errors: 0 (CRC) 0 (Timeout) 0 (Page Fault) │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ PCIe / NVLink │ +│ ├─ Link: Gen4 x16 │ Throughput: 12.5 GB/s / 31.5 GB/s (40%) │ +│ ├─ Replay Timer: 0 │ NAK: 0 │ L0→L1: 12 L1→L0: 12 │ +│ └─ NVLink: N/A (single GPU) │ +└─────────────────────────────────────────────────────────────────────────────┘ + [F1] Help [F2] GPU Select [F3] Engine Detail [F4] Memory Detail [F5] Processes [F6] GSP [F10] Quit +``` + +## Key Bindings +| Key | Action | +|-----|--------| +| `↑/↓` | Navigate process list | +| `←/→` | Switch GPU (multi-GPU) | +| `F1` | Help | +| `F2` | GPU Select menu | +| `F3` | Engine detail view | +| `F4` | Memory detail view | +| `F5` | Process list (expanded) | +| `F6` | GSP-RM telemetry | +| `F7` | PCIe/NVLink detail | +| `F8` | Power/thermal graphs | +| `F9` | Configuration | +| `F10` / `q` | Quit | +| `Space` | Pause/resume updates | +| `s` | Change sample interval (100ms-10s) | +| `u` | Sort by GPU util / Memory / PID | +| `k` | Kill process (send SIGTERM to guest) | +| `e` | Export snapshot (JSON) | + +## Engine Detail View (F3) +``` +┌─ GR Engine Detail ────────────────────────────────────────────────────────┐ +│ Channel 0 (PID 1247, CTX 0x3) │ Util: 82% │ Submitted: 1.2M ops │ +│ ├─ Pushbuffer: 0x7f8a... │ Fence: 0x123456 │ Pending: 47 │ +│ ├─ Context: 0x3 │ Page Tables: 0x100000 │ VRAM: 2.1G │ +│ └─ Perf: SM=82% TEX=45% L2=67% ROPS=34% │ +│ Channel 3 (PID 1247, CTX 0x3) │ Util: 12% │ Submitted: 45K ops │ +│ └─ ... │ +│ Channel 7 (PID 3891, CTX 0x7) │ Util: 5% │ Submitted: 12K ops │ +│ └─ ... │ +│ Channel 12 (PID 3891, CTX 0x7) │ Util: 0% │ Submitted: 0 ops │ +│ └─ IDLE │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Memory Detail View (F4) +``` +┌─ Memory Detail ────────────────────────────────────────────────────────────┐ +│ VRAM Allocations (by PID) │ +│ ├─ 1247: 18.2G ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │ +│ ├─ 3891: 0.6G ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% │ +│ └─ System: 0.1G ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │ +│ │ +│ Fragmentation: 12% │ Largest Free Block: 2.1G │ Allocation Count: 347 │ +│ │ +│ BAR1 Mappings (CPU→GPU) │ +│ ├─ 0x7f8a00000000 → GPU VA 0x100000 (256M) RW PID 1247 │ +│ └─ 0x7f8b00000000 → GPU VA 0x200000 (128M) RO PID 3891 │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## GSP-RM Telemetry View (F6) +``` +┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐ +│ RPC Queues (depth / max=64) │ +│ ├─ GR: ████████████░░░░░░░░░░ 12/64 │ Avg Latency: 2.3μs │ +│ ├─ CE0: ███░░░░░░░░░░░░░░░░░░░░░ 3/64 │ Avg Latency: 1.1μs │ +│ ├─ CE1: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ +│ ├─ NVDEC: ██████████████████████ 64/64 │ Avg Latency: 4.7μs ⚠ SATURATED│ +│ ├─ NVENC: ████████████░░░░░░░░░░░░░ 1/64 │ Avg Latency: 3.2μs │ +│ └─ OFA: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ +│ │ +│ Doorbell Statistics │ +│ ├─ Total Rate: 45,231/s │ Peak: 120,451/s │ +│ ├─ GR: 28,123/s │ CE: 8,456/s │ DEC: 5,234/s │ ENC: 3,418/s │ +│ └─ Errors: 0 (spurious) 0 (invalid) │ +│ │ +│ Firmware Threads (16 total) │ +│ ├─ Active: 8 │ Idle: 6 │ Blocked: 2 │ +│ ├─ Thread 0: GR Scheduler │ CPU: 45% │ RPC: 12,341/s │ +│ ├─ Thread 1: CE Manager │ CPU: 12% │ RPC: 3,456/s │ +│ ├─ Thread 2: DEC Manager │ CPU: 78% │ RPC: 5,234/s ⚠ HIGH │ +│ └─ ... │ +│ │ +│ WPR State │ +│ ├─ Region 0: 0x100000-0x200000 (1M) READ-ONLY PID 1247 │ +│ ├─ Region 1: 0x300000-0x400000 (1M) NO-ACCESS PID 1247 │ +│ └─ Region 2: 0x500000-0x600000 (1M) READ-WRITE PID 3891 │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Multi-GPU Support +``` +rmtop -g 0 # Start on GPU 0 +rmtop -g all # Cycle through all GPUs (tab) +rmtop --layout=grid # Show all GPUs in grid (requires wide terminal) +``` + +## Configuration (`~/.config/rmtop/config.toml`) +```toml +[general] +sample_interval_ms = 500 +theme = "dark" # dark, light, mono +show_gsp = true +show_pcie = true +show_nvlink = false + +[colors] +engine_high = "red" +engine_medium = "yellow" +engine_low = "green" +warning = "red" +critical = "bright_red" + +[columns] +# Show/hide columns in process list +pid = true +name = true +engine = true +context = true +vram = true +sysmem = true +util = true +state = true + +[alerts] +gsp_queue_warning = 48 # 75% of max +gsp_queue_critical = 56 # 87.5% of max +temp_warning = 80 +temp_critical = 85 +power_warning = 400 +``` + +## Build +```bash +cd open-iov/tools/rmtop +make +# Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess +``` + +## Architecture +``` +rmtop/ +├── src/ +│ ├── main.c # ncurses init, main loop +│ ├── ui.c # Layout, rendering, input handling +│ ├── gpu_panel.c # GPU summary panel +│ ├── engine_panel.c # Engine utilization bars +│ ├── memory_panel.c # Memory breakdown +│ ├── process_panel.c # Process/channel list +│ ├── power_panel.c # Power/clocks/thermals +│ ├── gsp_panel.c # GSP-RM telemetry +│ ├── pcie_panel.c # PCIe/NVLink +│ ├── detail_views.c # F3-F8 detail panels +│ ├── sampler.c # Background sampling thread +│ └── config.c # Config file parsing +├── include/ +│ └── rmtop.h +├── lib/ +│ ├── librmapi.a +│ └── libgpumon.a +└── Makefile +``` + +## Future Extensions +- **Remote Dashboard**: WebSocket server for browser-based monitoring +- **Historical Graphs**: In-terminal sparklines for utilization history +- **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO +- **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads +- **Policy Visualization**: Show resserve reservations, quotas +- **Migration Tracking**: Live migration progress bars +- **Telemetry Export**: Background Prometheus/OpenTelemetry push \ No newline at end of file -- cgit v1.2.3