# rmtop - GPU htop Equivalent ## Overview `rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry. ## Source Code Reuse Same recycled components as `rmtool`: - `gpumon_perfmon` - sampling infrastructure, perfmon counters - `nvidia-smi` - GPU enumeration, NVML-style queries - `nv2080` - direct RM control for real-time state - `resserve` - resource server state - `resman` - resource manager heaps/clients ## UI Layout ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ rmtop 1.0 │ GPU 0: RTX 5090 (GB202) │ Driver: 610.XX │ 14:32:15 │ ├─────────────────────────────────────────────────────────────────────────────┤ │ ENGINES │ UTIL │ FREQ │ POWER │ TEMP │ ACTIVE CHANNELS │ │ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │ 320W │ 78°C │ ch: 0,3,7,12 │ │ ├─ CE0 (Copy) │ ████░░░░░░ 41% │ 2100MHz │ │ │ ch: 1,4 │ │ ├─ CE1 (Copy) │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ │ ├─ NVDEC0 │ ██████████ 100% │ 1800MHz │ │ │ ch: 2 │ │ ├─ NVDEC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ │ ├─ NVENC0 │ ████████░░ 82% │ 1800MHz │ │ │ ch: 5 │ │ ├─ NVENC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ │ ├─ NVJPG │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ │ └─ OFA │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ ├─────────────────────────────────────────────────────────────────────────────┤ │ MEMORY │ USED │ TOTAL │ FREE │ FRAG │ BAR1 │ │ ├─ VRAM │ 18.2G │ 24.0G │ 5.8G │ 12% │ 256M / 256M │ │ ├─ System RAM │ 4.1G │ 64.0G │ 59.9G │ 3% │ │ │ └─ BAR1 (CPU) │ 12M │ 256M │ 244M │ 1% │ │ ├─────────────────────────────────────────────────────────────────────────────┤ │ CHANNELS / CONTEXTS (PID │ ENG │ CTX │ VRAM │ SYSMEM │ UTIL │ STATE) │ │ ├─ 1247 │ GR │ 0x3 │ 2.1G │ 512M │ 82% │ RUNNING │ │ ├─ 1247 │ CE0 │ 0x3 │ 128M │ 64M │ 41% │ RUNNING │ │ ├─ 1247 │ DEC │ 0x3 │ 4.0G │ 256M │ 100% │ RUNNING │ │ ├─ 1247 │ ENC │ 0x3 │ 1.5G │ 128M │ 82% │ RUNNING │ │ ├─ 3891 │ GR │ 0x7 │ 512M │ 128M │ 12% │ WAITING │ │ └─ 3891 │ CE0 │ 0x7 │ 64M │ 32M │ 5% │ WAITING │ ├─────────────────────────────────────────────────────────────────────────────┤ │ POWER / CLOCKS / THERMALS │ │ ├─ Power Draw: 320W / 450W (71%) │ GPU Clock: 2850 MHz (max) │ │ ├─ Mem Clock: 1313 MHz (max) │ SM Clock: 2850 MHz │ │ ├─ Temp: 78°C (slowdown: 83°C) │ Fan: 65% (auto) │ │ └─ PState: P0 (Performance) │ Perf Cap: None │ ├─────────────────────────────────────────────────────────────────────────────┤ │ GSP-RM TELEMETRY │ │ ├─ RPC Queue: GR=12 CE=3 DEC=0 ENC=1 (depth / max=64) │ │ ├─ Doorbell Rate: 45K/s │ Firmware Threads: 8/16 active │ │ ├─ WPR Regions: 3 active │ Migration: IDLE │ │ └─ Errors: 0 (CRC) 0 (Timeout) 0 (Page Fault) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ PCIe / NVLink │ │ ├─ Link: Gen4 x16 │ Throughput: 12.5 GB/s / 31.5 GB/s (40%) │ │ ├─ Replay Timer: 0 │ NAK: 0 │ L0→L1: 12 L1→L0: 12 │ │ └─ NVLink: N/A (single GPU) │ └─────────────────────────────────────────────────────────────────────────────┘ [F1] Help [F2] GPU Select [F3] Engine Detail [F4] Memory Detail [F5] Processes [F6] GSP [F10] Quit ``` ## Key Bindings | Key | Action | |-----|--------| | `↑/↓` | Navigate process list | | `←/→` | Switch GPU (multi-GPU) | | `F1` | Help | | `F2` | GPU Select menu | | `F3` | Engine detail view | | `F4` | Memory detail view | | `F5` | Process list (expanded) | | `F6` | GSP-RM telemetry | | `F7` | PCIe/NVLink detail | | `F8` | Power/thermal graphs | | `F9` | Configuration | | `F10` / `q` | Quit | | `Space` | Pause/resume updates | | `s` | Change sample interval (100ms-10s) | | `u` | Sort by GPU util / Memory / PID | | `k` | Kill process (send SIGTERM to guest) | | `e` | Export snapshot (JSON) | ## Engine Detail View (F3) ``` ┌─ GR Engine Detail ────────────────────────────────────────────────────────┐ │ Channel 0 (PID 1247, CTX 0x3) │ Util: 82% │ Submitted: 1.2M ops │ │ ├─ Pushbuffer: 0x7f8a... │ Fence: 0x123456 │ Pending: 47 │ │ ├─ Context: 0x3 │ Page Tables: 0x100000 │ VRAM: 2.1G │ │ └─ Perf: SM=82% TEX=45% L2=67% ROPS=34% │ │ Channel 3 (PID 1247, CTX 0x3) │ Util: 12% │ Submitted: 45K ops │ │ └─ ... │ │ Channel 7 (PID 3891, CTX 0x7) │ Util: 5% │ Submitted: 12K ops │ │ └─ ... │ │ Channel 12 (PID 3891, CTX 0x7) │ Util: 0% │ Submitted: 0 ops │ │ └─ IDLE │ └────────────────────────────────────────────────────────────────────────────────┘ ``` ## Memory Detail View (F4) ``` ┌─ Memory Detail ────────────────────────────────────────────────────────────┐ │ VRAM Allocations (by PID) │ │ ├─ 1247: 18.2G ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │ │ ├─ 3891: 0.6G ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% │ │ └─ System: 0.1G ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │ │ │ │ Fragmentation: 12% │ Largest Free Block: 2.1G │ Allocation Count: 347 │ │ │ │ BAR1 Mappings (CPU→GPU) │ │ ├─ 0x7f8a00000000 → GPU VA 0x100000 (256M) RW PID 1247 │ │ └─ 0x7f8b00000000 → GPU VA 0x200000 (128M) RO PID 3891 │ └────────────────────────────────────────────────────────────────────────────────┘ ``` ## GSP-RM Telemetry View (F6) ``` ┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐ │ RPC Queues (depth / max=64) │ │ ├─ GR: ████████████░░░░░░░░░░ 12/64 │ Avg Latency: 2.3μs │ │ ├─ CE0: ███░░░░░░░░░░░░░░░░░░░░░ 3/64 │ Avg Latency: 1.1μs │ │ ├─ CE1: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ │ ├─ NVDEC: ██████████████████████ 64/64 │ Avg Latency: 4.7μs ⚠ SATURATED│ │ ├─ NVENC: ████████████░░░░░░░░░░░░░ 1/64 │ Avg Latency: 3.2μs │ │ └─ OFA: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ │ │ │ Doorbell Statistics │ │ ├─ Total Rate: 45,231/s │ Peak: 120,451/s │ │ ├─ GR: 28,123/s │ CE: 8,456/s │ DEC: 5,234/s │ ENC: 3,418/s │ │ └─ Errors: 0 (spurious) 0 (invalid) │ │ │ │ Firmware Threads (16 total) │ │ ├─ Active: 8 │ Idle: 6 │ Blocked: 2 │ │ ├─ Thread 0: GR Scheduler │ CPU: 45% │ RPC: 12,341/s │ │ ├─ Thread 1: CE Manager │ CPU: 12% │ RPC: 3,456/s │ │ ├─ Thread 2: DEC Manager │ CPU: 78% │ RPC: 5,234/s ⚠ HIGH │ │ └─ ... │ │ │ │ WPR State │ │ ├─ Region 0: 0x100000-0x200000 (1M) READ-ONLY PID 1247 │ │ ├─ Region 1: 0x300000-0x400000 (1M) NO-ACCESS PID 1247 │ │ └─ Region 2: 0x500000-0x600000 (1M) READ-WRITE PID 3891 │ └────────────────────────────────────────────────────────────────────────────────┘ ``` ## Multi-GPU Support ``` rmtop -g 0 # Start on GPU 0 rmtop -g all # Cycle through all GPUs (tab) rmtop --layout=grid # Show all GPUs in grid (requires wide terminal) ``` ## Configuration (`~/.config/rmtop/config.toml`) ```toml [general] sample_interval_ms = 500 theme = "dark" # dark, light, mono show_gsp = true show_pcie = true show_nvlink = false [colors] engine_high = "red" engine_medium = "yellow" engine_low = "green" warning = "red" critical = "bright_red" [columns] # Show/hide columns in process list pid = true name = true engine = true context = true vram = true sysmem = true util = true state = true [alerts] gsp_queue_warning = 48 # 75% of max gsp_queue_critical = 56 # 87.5% of max temp_warning = 80 temp_critical = 85 power_warning = 400 ``` ## Build ```bash cd open-iov/tools/rmtop make # Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess ``` ## Architecture ``` rmtop/ ├── src/ │ ├── main.c # ncurses init, main loop │ ├── ui.c # Layout, rendering, input handling │ ├── gpu_panel.c # GPU summary panel │ ├── engine_panel.c # Engine utilization bars │ ├── memory_panel.c # Memory breakdown │ ├── process_panel.c # Process/channel list │ ├── power_panel.c # Power/clocks/thermals │ ├── gsp_panel.c # GSP-RM telemetry │ ├── pcie_panel.c # PCIe/NVLink │ ├── detail_views.c # F3-F8 detail panels │ ├── sampler.c # Background sampling thread │ └── config.c # Config file parsing ├── include/ │ └── rmtop.h ├── lib/ │ ├── librmapi.a │ └── libgpumon.a └── Makefile ``` ## Future Extensions - **Remote Dashboard**: WebSocket server for browser-based monitoring - **Historical Graphs**: In-terminal sparklines for utilization history - **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO - **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads - **Policy Visualization**: Show resserve reservations, quotas - **Migration Tracking**: Live migration progress bars - **Telemetry Export**: Background Prometheus/OpenTelemetry push