summaryrefslogtreecommitdiffstats
path: root/docs/tools
diff options
context:
space:
mode:
authorOpen-IOV <dev@open-iov.org>2026-09-23 05:44:08 -0400
committerOpen-IOV <dev@open-iov.org>2026-09-23 05:46:43 -0400
commit68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 (patch)
tree3dc8ce2e6af313a97ec6852f5c663a9a93665d2e /docs/tools
downloadopen-iov-68cdd7728a7cc6715bbbff7decfe2b11c3bdf558.tar.gz
open-iov-68cdd7728a7cc6715bbbff7decfe2b11c3bdf558.zip
Open-IOV: initial public release
Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.
Diffstat (limited to 'docs/tools')
-rw-r--r--docs/tools/README.md80
-rw-r--r--docs/tools/rmtool.md150
-rw-r--r--docs/tools/rmtop.md227
3 files changed, 457 insertions, 0 deletions
diff --git a/docs/tools/README.md b/docs/tools/README.md
new file mode 100644
index 0000000..232fbe5
--- /dev/null
+++ b/docs/tools/README.md
@@ -0,0 +1,80 @@
+# Open-IOV Tools Documentation
+
+## Overview
+This directory contains documentation for userspace tools built on top of the
+Open-IOV vGPU-IO stack. The RM-API tooling reuses behaviour and interfaces
+from the driver components we studied by reverse engineering and decompilation
+(tracked docs describe components and behaviour).
+
+## Tools
+
+### Debugging & Observability (Implemented)
+| Tool | Location | Description |
+|------|----------|-------------|
+| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output |
+| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations |
+| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner |
+| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) |
+| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) |
+| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM |
+
+### RM API Tooling (Planned)
+| Tool | Documentation | Description |
+|------|---------------|-------------|
+| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control |
+| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard |
+
+## Code Reuse Sources (component level)
+
+All planned tools reuse the driver components we studied. Tracked
+documentation refers to them by component only:
+
+| Component | Use Case |
+|-----------|----------|
+| NVML / nvidia-smi component | CLI framework, GPU enumeration, NV2080 ioctls |
+| gpumon / perfmon component | Performance sampling, perfmon counters |
+| NV2080 control component | RM control interfaces (classes, methods, events) |
+| Resource server component | Resource Server (resserve) API |
+| CUDA software (CUDASW) component | CUDA internal layer behaviours |
+| Resource manager component | Resman API |
+
+## Build Integration
+Tools will be built under `open-iov/tools/`:
+```
+tools/
+├── rmtool/
+│ ├── src/
+│ ├── lib/
+│ └── Makefile
+├── rmtop/
+│ ├── src/
+│ ├── lib/
+│ └── Makefile
+└── Makefile # Builds all tools
+```
+
+## Dependencies
+- `libnvidia-rmapi.so` - reused RM API layer (component level)
+- `libgpumon.so` - reused perfmon layer (component level)
+- `libpciaccess` - PCI configuration access
+- `libnuma` - NUMA topology
+- `ncursesw` - rmtop UI (wide character support)
+- `libjson-c` - JSON output formatting
+- `libcurl` - Remote API (future)
+
+## Integration with Open-IOV Stack
+Tools will integrate with the host VFIO driver via:
+- `/dev/nvidia*` character devices (existing nvidia.ko)
+- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`)
+- VFIO ioctls for guest state introspection
+- GSP-RM telemetry via firmware analyzer / dynamic dispatcher
+
+## Future Tool Ideas
+| Tool | Purpose |
+|------|---------|
+| `rmmigrate` | Live migration coordinator via resserve |
+| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) |
+| `rmpolicy` | Policy engine for automated engine allocation |
+| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) |
+| `rmguest` | Guest introspection via vGPU VFIO interface |
+| `rmfirmware` | GSP firmware analysis / update utility | \ No newline at end of file
diff --git a/docs/tools/rmtool.md b/docs/tools/rmtool.md
new file mode 100644
index 0000000..a7ec2c6
--- /dev/null
+++ b/docs/tools/rmtool.md
@@ -0,0 +1,150 @@
+# rmtool - RM API Swiss Army Knife
+
+## Overview
+`rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built on top of driver components studied through reverse engineering and
+decompilation (referenced by component).
+
+## Source Code Reuse
+| Component | Source Location | Purpose |
+|-----------|-----------------|---------|
+| nvidia-smi component | CLI framework, NV2080 ioctls, GPU enumeration |
+| gpumon / perfmon component | Performance monitoring, sampling infrastructure |
+| NV2080 control component | RM control interfaces (classes, methods, events) |
+| Resource server component | Resource Server (resserve) API |
+| CUDA software (CUDASW) component | CUDA internal layer |
+| Resource manager component | Resman API |
+
+## Core Commands
+
+### GPU Enumeration & Topology
+```bash
+rmtool gpu list # List all GPUs with PCI info
+rmtool gpu topology # Show GPU topology (NVLink, PCIe, PCI IDs)
+rmtool gpu info -d 0000:65:00.0 # Detailed GPU info (like nvidia-smi -q)
+```
+
+### Engine Management
+```bash
+rmtool engine list # List all engines (GR, CE, NVDEC, NVENC, NVJPG, OFA)
+rmtool engine info -d 0 -e gr # Engine details (registers, capabilities)
+rmtool engine allocate -d 0 -e gr -c 4 # Allocate 4 GR channels
+rmtool engine start -d 0 -e gr -c 0 # Start engine channel
+rmtool engine stop -d 0 -e gr -c 0 # Stop engine channel
+rmtool engine config -d 0 -e nvenc --preset=h264 --bitrate=10M
+```
+
+### Channel & Context Control
+```bash
+rmtool channel list -d 0 # List all channels on GPU
+rmtool channel create -d 0 -e gr -t compute # Create compute channel
+rmtool channel submit -d 0 -c 0 --pushbuf=work.bin # Submit work
+rmtool channel status -d 0 -c 0 # Channel status (busy, pending, error)
+rmtool context list -d 0 # List all contexts
+rmtool context info -d 0 -c 0 # Context details (page tables, memory)
+```
+
+### Memory Operations
+```bash
+rmtool mem alloc -d 0 --size=1G --type=video # Allocate VRAM
+rmtool mem alloc -d 0 --size=512M --type=system # Allocate sysmem
+rmtool mem map -d 0 -h 0x1234 --gpu-va=0x100000 # Map to GPU VA
+rmtool mem unmap -d 0 -h 0x1234 # Unmap
+rmtool mem pin -d 0 -h 0x1234 # Pin for DMA
+rmtool mem info -d 0 -h 0x1234 # Allocation details
+rmtool mem stats -d 0 # VRAM/sysmem usage breakdown
+```
+
+### Performance Sampling
+```bash
+rmtool sample engine -d 0 -e gr --interval=100ms --count=100
+rmtool sample clocks -d 0 --interval=1s
+rmtool sample power -d 0 --interval=1s
+rmtool sample thermal -d 0 --interval=1s
+rmtool sample pcie -d 0 --interval=1s
+```
+
+### NV2080 Direct Control
+```bash
+rmtool nv2080 class list -d 0 # List all NV2080 classes
+rmtool nv2080 method call -d 0 -c 0x906D -m 0x100 --params=...
+rmtool nv2080 event subscribe -d 0 -c 0x906D -e 0x100
+rmtool nv2080 event wait -d 0 -c 0x906D -e 0x100 --timeout=5000
+```
+
+### Resource Server (resserve) Integration
+```bash
+rmtool resserve status # Resource Server status
+rmtool resserve reservation list # List all reservations
+rmtool resserve reservation create --gpu=0 --engine=gr --size=4
+rmtool resserve reservation release --id=123
+```
+
+### CUDA Internal Layer (CUDASW)
+```bash
+rmtool cudasw context list # CUDA contexts
+rmtool cudasw module list # Loaded CUDA modules
+rmtool cudasw stream list -c 0 # Streams in context
+rmtool cudasw event list -c 0 # Events in context
+```
+
+### Resource Manager (Resman)
+```bash
+rmtool resman client list # Resman clients
+rmtool resman resource list -c 0 # Resources owned by client
+rmtool resman heap info -d 0 # Heap state
+```
+
+### GSP-RM Telemetry
+```bash
+rmtool gsp rpc-stats -d 0 # RPC queue depths, latency
+rmtool gsp doorbell -d 0 # Doorbell rates
+rmtool gsp firmware -d 0 # Firmware version, threads
+rmtool gsp wpr -d 0 # WPR state
+```
+
+## Output Formats
+```bash
+rmtool gpu list --format=json
+rmtool engine list --format=csv
+rmtool sample engine --format=prometheus
+rmtool mem stats --format=influx
+```
+
+## Build
+```bash
+cd open-iov/tools/rmtool
+make
+# Links against: libnvidia-rmapi.so (reused RM API component), libpciaccess, libnuma
+```
+
+## Architecture
+```
+rmtool/
+├── src/
+│ ├── main.c # CLI parsing, command dispatch
+│ ├── gpu.c # GPU enumeration, topology
+│ ├── engine.c # Engine allocation/control
+│ ├── channel.c # Channel/context management
+│ ├── memory.c # Memory alloc/map/pin
+│ ├── sample.c # Performance sampling
+│ ├── nv2080.c # NV2080 direct control
+│ ├── resserve.c # Resource Server client
+│ ├── cudasw.c # CUDA Internal Layer
+│ ├── resman.c # Resource Manager
+│ ├── gsp.c # GSP-RM telemetry
+│ └── output.c # JSON/CSV/Prometheus formatters
+├── include/
+│ └── rmtool.h # Internal headers
+├── lib/
+│ ├── librmapi.a # Recycled RM API code
+│ └── libgpumon.a # Recycled perfmon code
+└── Makefile
+```
+
+## Future Extensions
+- **Remote gRPC/REST API** for fleet management
+- **Policy Engine** for automated engine allocation
+- **Migration Coordination** via resserve
+- **Telemetry Export** (Prometheus, OpenTelemetry, InfluxDB)
+- **Guest Introspection** via vGPU VFIO interface
+- **Workload Profiling** with CUDA/GR engine correlation \ No newline at end of file
diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md
new file mode 100644
index 0000000..6408268
--- /dev/null
+++ b/docs/tools/rmtop.md
@@ -0,0 +1,227 @@
+# rmtop - GPU htop Equivalent
+
+## Overview
+`rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry.
+
+## Source Code Reuse
+Same recycled components as `rmtool`:
+- `gpumon_perfmon` - sampling infrastructure, perfmon counters
+- `nvidia-smi` - GPU enumeration, NVML-style queries
+- `nv2080` - direct RM control for real-time state
+- `resserve` - resource server state
+- `resman` - resource manager heaps/clients
+
+## UI Layout
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ rmtop 1.0 │ GPU 0: RTX 5090 (GB202) │ Driver: 610.XX │ 14:32:15 │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ ENGINES │ UTIL │ FREQ │ POWER │ TEMP │ ACTIVE CHANNELS │
+│ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │ 320W │ 78°C │ ch: 0,3,7,12 │
+│ ├─ CE0 (Copy) │ ████░░░░░░ 41% │ 2100MHz │ │ │ ch: 1,4 │
+│ ├─ CE1 (Copy) │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ ├─ NVDEC0 │ ██████████ 100% │ 1800MHz │ │ │ ch: 2 │
+│ ├─ NVDEC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ ├─ NVENC0 │ ████████░░ 82% │ 1800MHz │ │ │ ch: 5 │
+│ ├─ NVENC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ ├─ NVJPG │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ └─ OFA │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ MEMORY │ USED │ TOTAL │ FREE │ FRAG │ BAR1 │
+│ ├─ VRAM │ 18.2G │ 24.0G │ 5.8G │ 12% │ 256M / 256M │
+│ ├─ System RAM │ 4.1G │ 64.0G │ 59.9G │ 3% │ │
+│ └─ BAR1 (CPU) │ 12M │ 256M │ 244M │ 1% │ │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ CHANNELS / CONTEXTS (PID │ ENG │ CTX │ VRAM │ SYSMEM │ UTIL │ STATE) │
+│ ├─ 1247 │ GR │ 0x3 │ 2.1G │ 512M │ 82% │ RUNNING │
+│ ├─ 1247 │ CE0 │ 0x3 │ 128M │ 64M │ 41% │ RUNNING │
+│ ├─ 1247 │ DEC │ 0x3 │ 4.0G │ 256M │ 100% │ RUNNING │
+│ ├─ 1247 │ ENC │ 0x3 │ 1.5G │ 128M │ 82% │ RUNNING │
+│ ├─ 3891 │ GR │ 0x7 │ 512M │ 128M │ 12% │ WAITING │
+│ └─ 3891 │ CE0 │ 0x7 │ 64M │ 32M │ 5% │ WAITING │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ POWER / CLOCKS / THERMALS │
+│ ├─ Power Draw: 320W / 450W (71%) │ GPU Clock: 2850 MHz (max) │
+│ ├─ Mem Clock: 1313 MHz (max) │ SM Clock: 2850 MHz │
+│ ├─ Temp: 78°C (slowdown: 83°C) │ Fan: 65% (auto) │
+│ └─ PState: P0 (Performance) │ Perf Cap: None │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ GSP-RM TELEMETRY │
+│ ├─ RPC Queue: GR=12 CE=3 DEC=0 ENC=1 (depth / max=64) │
+│ ├─ Doorbell Rate: 45K/s │ Firmware Threads: 8/16 active │
+│ ├─ WPR Regions: 3 active │ Migration: IDLE │
+│ └─ Errors: 0 (CRC) 0 (Timeout) 0 (Page Fault) │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ PCIe / NVLink │
+│ ├─ Link: Gen4 x16 │ Throughput: 12.5 GB/s / 31.5 GB/s (40%) │
+│ ├─ Replay Timer: 0 │ NAK: 0 │ L0→L1: 12 L1→L0: 12 │
+│ └─ NVLink: N/A (single GPU) │
+└─────────────────────────────────────────────────────────────────────────────┘
+ [F1] Help [F2] GPU Select [F3] Engine Detail [F4] Memory Detail [F5] Processes [F6] GSP [F10] Quit
+```
+
+## Key Bindings
+| Key | Action |
+|-----|--------|
+| `↑/↓` | Navigate process list |
+| `←/→` | Switch GPU (multi-GPU) |
+| `F1` | Help |
+| `F2` | GPU Select menu |
+| `F3` | Engine detail view |
+| `F4` | Memory detail view |
+| `F5` | Process list (expanded) |
+| `F6` | GSP-RM telemetry |
+| `F7` | PCIe/NVLink detail |
+| `F8` | Power/thermal graphs |
+| `F9` | Configuration |
+| `F10` / `q` | Quit |
+| `Space` | Pause/resume updates |
+| `s` | Change sample interval (100ms-10s) |
+| `u` | Sort by GPU util / Memory / PID |
+| `k` | Kill process (send SIGTERM to guest) |
+| `e` | Export snapshot (JSON) |
+
+## Engine Detail View (F3)
+```
+┌─ GR Engine Detail ────────────────────────────────────────────────────────┐
+│ Channel 0 (PID 1247, CTX 0x3) │ Util: 82% │ Submitted: 1.2M ops │
+│ ├─ Pushbuffer: 0x7f8a... │ Fence: 0x123456 │ Pending: 47 │
+│ ├─ Context: 0x3 │ Page Tables: 0x100000 │ VRAM: 2.1G │
+│ └─ Perf: SM=82% TEX=45% L2=67% ROPS=34% │
+│ Channel 3 (PID 1247, CTX 0x3) │ Util: 12% │ Submitted: 45K ops │
+│ └─ ... │
+│ Channel 7 (PID 3891, CTX 0x7) │ Util: 5% │ Submitted: 12K ops │
+│ └─ ... │
+│ Channel 12 (PID 3891, CTX 0x7) │ Util: 0% │ Submitted: 0 ops │
+│ └─ IDLE │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Memory Detail View (F4)
+```
+┌─ Memory Detail ────────────────────────────────────────────────────────────┐
+│ VRAM Allocations (by PID) │
+│ ├─ 1247: 18.2G ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │
+│ ├─ 3891: 0.6G ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% │
+│ └─ System: 0.1G ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │
+│ │
+│ Fragmentation: 12% │ Largest Free Block: 2.1G │ Allocation Count: 347 │
+│ │
+│ BAR1 Mappings (CPU→GPU) │
+│ ├─ 0x7f8a00000000 → GPU VA 0x100000 (256M) RW PID 1247 │
+│ └─ 0x7f8b00000000 → GPU VA 0x200000 (128M) RO PID 3891 │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## GSP-RM Telemetry View (F6)
+```
+┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐
+│ RPC Queues (depth / max=64) │
+│ ├─ GR: ████████████░░░░░░░░░░ 12/64 │ Avg Latency: 2.3μs │
+│ ├─ CE0: ███░░░░░░░░░░░░░░░░░░░░░ 3/64 │ Avg Latency: 1.1μs │
+│ ├─ CE1: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │
+│ ├─ NVDEC: ██████████████████████ 64/64 │ Avg Latency: 4.7μs ⚠ SATURATED│
+│ ├─ NVENC: ████████████░░░░░░░░░░░░░ 1/64 │ Avg Latency: 3.2μs │
+│ └─ OFA: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │
+│ │
+│ Doorbell Statistics │
+│ ├─ Total Rate: 45,231/s │ Peak: 120,451/s │
+│ ├─ GR: 28,123/s │ CE: 8,456/s │ DEC: 5,234/s │ ENC: 3,418/s │
+│ └─ Errors: 0 (spurious) 0 (invalid) │
+│ │
+│ Firmware Threads (16 total) │
+│ ├─ Active: 8 │ Idle: 6 │ Blocked: 2 │
+│ ├─ Thread 0: GR Scheduler │ CPU: 45% │ RPC: 12,341/s │
+│ ├─ Thread 1: CE Manager │ CPU: 12% │ RPC: 3,456/s │
+│ ├─ Thread 2: DEC Manager │ CPU: 78% │ RPC: 5,234/s ⚠ HIGH │
+│ └─ ... │
+│ │
+│ WPR State │
+│ ├─ Region 0: 0x100000-0x200000 (1M) READ-ONLY PID 1247 │
+│ ├─ Region 1: 0x300000-0x400000 (1M) NO-ACCESS PID 1247 │
+│ └─ Region 2: 0x500000-0x600000 (1M) READ-WRITE PID 3891 │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Multi-GPU Support
+```
+rmtop -g 0 # Start on GPU 0
+rmtop -g all # Cycle through all GPUs (tab)
+rmtop --layout=grid # Show all GPUs in grid (requires wide terminal)
+```
+
+## Configuration (`~/.config/rmtop/config.toml`)
+```toml
+[general]
+sample_interval_ms = 500
+theme = "dark" # dark, light, mono
+show_gsp = true
+show_pcie = true
+show_nvlink = false
+
+[colors]
+engine_high = "red"
+engine_medium = "yellow"
+engine_low = "green"
+warning = "red"
+critical = "bright_red"
+
+[columns]
+# Show/hide columns in process list
+pid = true
+name = true
+engine = true
+context = true
+vram = true
+sysmem = true
+util = true
+state = true
+
+[alerts]
+gsp_queue_warning = 48 # 75% of max
+gsp_queue_critical = 56 # 87.5% of max
+temp_warning = 80
+temp_critical = 85
+power_warning = 400
+```
+
+## Build
+```bash
+cd open-iov/tools/rmtop
+make
+# Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess
+```
+
+## Architecture
+```
+rmtop/
+├── src/
+│ ├── main.c # ncurses init, main loop
+│ ├── ui.c # Layout, rendering, input handling
+│ ├── gpu_panel.c # GPU summary panel
+│ ├── engine_panel.c # Engine utilization bars
+│ ├── memory_panel.c # Memory breakdown
+│ ├── process_panel.c # Process/channel list
+│ ├── power_panel.c # Power/clocks/thermals
+│ ├── gsp_panel.c # GSP-RM telemetry
+│ ├── pcie_panel.c # PCIe/NVLink
+│ ├── detail_views.c # F3-F8 detail panels
+│ ├── sampler.c # Background sampling thread
+│ └── config.c # Config file parsing
+├── include/
+│ └── rmtop.h
+├── lib/
+│ ├── librmapi.a
+│ └── libgpumon.a
+└── Makefile
+```
+
+## Future Extensions
+- **Remote Dashboard**: WebSocket server for browser-based monitoring
+- **Historical Graphs**: In-terminal sparklines for utilization history
+- **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO
+- **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads
+- **Policy Visualization**: Show resserve reservations, quotas
+- **Migration Tracking**: Live migration progress bars
+- **Telemetry Export**: Background Prometheus/OpenTelemetry push \ No newline at end of file