summaryrefslogtreecommitdiffstats
path: root/docs/tools
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-14 06:43:11 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-14 06:43:11 -0400
commit5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60 (patch)
tree7523e29d8044d4d5b6dcd7c1c888a3491137fcd1 /docs/tools
parent9a7263fce4e3156349a526631805fce66b87fb1b (diff)
downloadopen-iov-5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60.tar.gz
open-iov-5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60.zip
VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl handlers
- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create() - Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions - Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP - Added config space read/write handlers at offset 0x70000000000 - Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers) - Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL) - Added udev rules for vfio group permissions - Created stack_manager.sh unified tool - Fixed mdev remove crash with deferred cleanup in vfio_device_release callback - Kernel 6.8+ IOMMUFD compatibility - QEMU launches successfully with vGPU passthrough
Diffstat (limited to 'docs/tools')
-rw-r--r--docs/tools/README.md75
-rw-r--r--docs/tools/rmtool.md149
-rw-r--r--docs/tools/rmtop.md227
3 files changed, 451 insertions, 0 deletions
diff --git a/docs/tools/README.md b/docs/tools/README.md
new file mode 100644
index 0000000..1c7c0b5
--- /dev/null
+++ b/docs/tools/README.md
@@ -0,0 +1,75 @@
+# Open-IOV Tools Documentation
+
+## Overview
+This directory contains documentation for userspace tools built on top of the Open-IOV vGPU-IO stack, leveraging recycled NVIDIA RM API code from `integ/gpu_drv/ext-tree/`.
+
+## Tools
+
+### Debugging & Observability (Implemented)
+| Tool | Location | Description |
+|------|----------|-------------|
+| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output |
+| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations |
+| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner |
+| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) |
+| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) |
+| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM |
+
+### RM API Tooling (Planned)
+| Tool | Documentation | Description |
+|------|---------------|-------------|
+| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control |
+| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard |
+
+## Code Reuse Sources
+All planned tools recycle existing NVIDIA driver code from `integ/gpu_drv/ext-tree/`:
+
+| Component | Path | Use Case |
+|-----------|------|----------|
+| nvidia-smi | `drivers/gpgpu/nvidia-smi/` | CLI framework, GPU enumeration, NV2080 ioctls |
+| gpumon/perfmon | `drivers/gpgpu/gpumon_perfmon/` | Performance sampling, perfmon counters |
+| NV2080 controls | `drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) |
+| Resource Server | `drivers/gpgpu/resserve/` | Resource Server (resserve) API |
+| CUDA Internal | `drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer |
+| Resource Manager | `drivers/gpgpu/resman/` | Resman API |
+
+## Build Integration
+Tools will be built under `open-iov/tools/`:
+```
+tools/
+├── rmtool/
+│ ├── src/
+│ ├── lib/
+│ └── Makefile
+├── rmtop/
+│ ├── src/
+│ ├── lib/
+│ └── Makefile
+└── Makefile # Builds all tools
+```
+
+## Dependencies
+- `libnvidia-rmapi.so` - Recycled RM API (from ext-tree)
+- `libgpumon.so` - Recycled perfmon (from ext-tree)
+- `libpciaccess` - PCI configuration access
+- `libnuma` - NUMA topology
+- `ncursesw` - rmtop UI (wide character support)
+- `libjson-c` - JSON output formatting
+- `libcurl` - Remote API (future)
+
+## Integration with Open-IOV Stack
+Tools will integrate with the host VFIO driver via:
+- `/dev/nvidia*` character devices (existing nvidia.ko)
+- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`)
+- VFIO ioctls for guest state introspection
+- GSP-RM telemetry via firmware analyzer / dynamic dispatcher
+
+## Future Tool Ideas
+| Tool | Purpose |
+|------|---------|
+| `rmmigrate` | Live migration coordinator via resserve |
+| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) |
+| `rmpolicy` | Policy engine for automated engine allocation |
+| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) |
+| `rmguest` | Guest introspection via vGPU VFIO interface |
+| `rmfirmware` | GSP firmware analysis / update utility | \ No newline at end of file
diff --git a/docs/tools/rmtool.md b/docs/tools/rmtool.md
new file mode 100644
index 0000000..e15306b
--- /dev/null
+++ b/docs/tools/rmtool.md
@@ -0,0 +1,149 @@
+# rmtool - RM API Swiss Army Knife
+
+## Overview
+`rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built by recycling existing code from the NVIDIA driver stack (`integ/gpu_drv/ext-tree/`).
+
+## Source Code Reuse
+| Component | Source Location | Purpose |
+|-----------|-----------------|---------|
+| nvidia-smi core | `ext-tree/drivers/gpgpu/nvidia-smi/` | CLI framework, NV2080 ioctls, GPU enumeration |
+| gpumon/perfmon | `ext-tree/drivers/gpgpu/gpumon_perfmon/` | Performance monitoring, sampling infrastructure |
+| NV2080 controls | `ext-tree/drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) |
+| Resource Server | `ext-tree/drivers/gpgpu/resserve/` | Resource Server (resserve) API |
+| CUDA Internal | `ext-tree/drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer |
+| Resource Manager | `ext-tree/drivers/gpgpu/resman/` | Resman API |
+
+## Core Commands
+
+### GPU Enumeration & Topology
+```bash
+rmtool gpu list # List all GPUs with PCI info
+rmtool gpu topology # Show GPU topology (NVLink, PCIe, PCI IDs)
+rmtool gpu info -d 0000:65:00.0 # Detailed GPU info (like nvidia-smi -q)
+```
+
+### Engine Management
+```bash
+rmtool engine list # List all engines (GR, CE, NVDEC, NVENC, NVJPG, OFA)
+rmtool engine info -d 0 -e gr # Engine details (registers, capabilities)
+rmtool engine allocate -d 0 -e gr -c 4 # Allocate 4 GR channels
+rmtool engine start -d 0 -e gr -c 0 # Start engine channel
+rmtool engine stop -d 0 -e gr -c 0 # Stop engine channel
+rmtool engine config -d 0 -e nvenc --preset=h264 --bitrate=10M
+```
+
+### Channel & Context Control
+```bash
+rmtool channel list -d 0 # List all channels on GPU
+rmtool channel create -d 0 -e gr -t compute # Create compute channel
+rmtool channel submit -d 0 -c 0 --pushbuf=work.bin # Submit work
+rmtool channel status -d 0 -c 0 # Channel status (busy, pending, error)
+rmtool context list -d 0 # List all contexts
+rmtool context info -d 0 -c 0 # Context details (page tables, memory)
+```
+
+### Memory Operations
+```bash
+rmtool mem alloc -d 0 --size=1G --type=video # Allocate VRAM
+rmtool mem alloc -d 0 --size=512M --type=system # Allocate sysmem
+rmtool mem map -d 0 -h 0x1234 --gpu-va=0x100000 # Map to GPU VA
+rmtool mem unmap -d 0 -h 0x1234 # Unmap
+rmtool mem pin -d 0 -h 0x1234 # Pin for DMA
+rmtool mem info -d 0 -h 0x1234 # Allocation details
+rmtool mem stats -d 0 # VRAM/sysmem usage breakdown
+```
+
+### Performance Sampling
+```bash
+rmtool sample engine -d 0 -e gr --interval=100ms --count=100
+rmtool sample clocks -d 0 --interval=1s
+rmtool sample power -d 0 --interval=1s
+rmtool sample thermal -d 0 --interval=1s
+rmtool sample pcie -d 0 --interval=1s
+```
+
+### NV2080 Direct Control
+```bash
+rmtool nv2080 class list -d 0 # List all NV2080 classes
+rmtool nv2080 method call -d 0 -c 0x906D -m 0x100 --params=...
+rmtool nv2080 event subscribe -d 0 -c 0x906D -e 0x100
+rmtool nv2080 event wait -d 0 -c 0x906D -e 0x100 --timeout=5000
+```
+
+### Resource Server (resserve) Integration
+```bash
+rmtool resserve status # Resource Server status
+rmtool resserve reservation list # List all reservations
+rmtool resserve reservation create --gpu=0 --engine=gr --size=4
+rmtool resserve reservation release --id=123
+```
+
+### CUDA Internal Layer (CUDASW)
+```bash
+rmtool cudasw context list # CUDA contexts
+rmtool cudasw module list # Loaded CUDA modules
+rmtool cudasw stream list -c 0 # Streams in context
+rmtool cudasw event list -c 0 # Events in context
+```
+
+### Resource Manager (Resman)
+```bash
+rmtool resman client list # Resman clients
+rmtool resman resource list -c 0 # Resources owned by client
+rmtool resman heap info -d 0 # Heap state
+```
+
+### GSP-RM Telemetry
+```bash
+rmtool gsp rpc-stats -d 0 # RPC queue depths, latency
+rmtool gsp doorbell -d 0 # Doorbell rates
+rmtool gsp firmware -d 0 # Firmware version, threads
+rmtool gsp wpr -d 0 # WPR state
+```
+
+## Output Formats
+```bash
+rmtool gpu list --format=json
+rmtool engine list --format=csv
+rmtool sample engine --format=prometheus
+rmtool mem stats --format=influx
+```
+
+## Build
+```bash
+cd open-iov/tools/rmtool
+make
+# Links against: libnvidia-rmapi.so (from ext-tree), libpciaccess, libnuma
+```
+
+## Architecture
+```
+rmtool/
+├── src/
+│ ├── main.c # CLI parsing, command dispatch
+│ ├── gpu.c # GPU enumeration, topology
+│ ├── engine.c # Engine allocation/control
+│ ├── channel.c # Channel/context management
+│ ├── memory.c # Memory alloc/map/pin
+│ ├── sample.c # Performance sampling
+│ ├── nv2080.c # NV2080 direct control
+│ ├── resserve.c # Resource Server client
+│ ├── cudasw.c # CUDA Internal Layer
+│ ├── resman.c # Resource Manager
+│ ├── gsp.c # GSP-RM telemetry
+│ └── output.c # JSON/CSV/Prometheus formatters
+├── include/
+│ └── rmtool.h # Internal headers
+├── lib/
+│ ├── librmapi.a # Recycled RM API code
+│ └── libgpumon.a # Recycled perfmon code
+└── Makefile
+```
+
+## Future Extensions
+- **Remote gRPC/REST API** for fleet management
+- **Policy Engine** for automated engine allocation
+- **Migration Coordination** via resserve
+- **Telemetry Export** (Prometheus, OpenTelemetry, InfluxDB)
+- **Guest Introspection** via vGPU VFIO interface
+- **Workload Profiling** with CUDA/GR engine correlation \ No newline at end of file
diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md
new file mode 100644
index 0000000..6408268
--- /dev/null
+++ b/docs/tools/rmtop.md
@@ -0,0 +1,227 @@
+# rmtop - GPU htop Equivalent
+
+## Overview
+`rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry.
+
+## Source Code Reuse
+Same recycled components as `rmtool`:
+- `gpumon_perfmon` - sampling infrastructure, perfmon counters
+- `nvidia-smi` - GPU enumeration, NVML-style queries
+- `nv2080` - direct RM control for real-time state
+- `resserve` - resource server state
+- `resman` - resource manager heaps/clients
+
+## UI Layout
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ rmtop 1.0 │ GPU 0: RTX 5090 (GB202) │ Driver: 610.XX │ 14:32:15 │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ ENGINES │ UTIL │ FREQ │ POWER │ TEMP │ ACTIVE CHANNELS │
+│ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │ 320W │ 78°C │ ch: 0,3,7,12 │
+│ ├─ CE0 (Copy) │ ████░░░░░░ 41% │ 2100MHz │ │ │ ch: 1,4 │
+│ ├─ CE1 (Copy) │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ ├─ NVDEC0 │ ██████████ 100% │ 1800MHz │ │ │ ch: 2 │
+│ ├─ NVDEC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ ├─ NVENC0 │ ████████░░ 82% │ 1800MHz │ │ │ ch: 5 │
+│ ├─ NVENC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ ├─ NVJPG │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+│ └─ OFA │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ MEMORY │ USED │ TOTAL │ FREE │ FRAG │ BAR1 │
+│ ├─ VRAM │ 18.2G │ 24.0G │ 5.8G │ 12% │ 256M / 256M │
+│ ├─ System RAM │ 4.1G │ 64.0G │ 59.9G │ 3% │ │
+│ └─ BAR1 (CPU) │ 12M │ 256M │ 244M │ 1% │ │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ CHANNELS / CONTEXTS (PID │ ENG │ CTX │ VRAM │ SYSMEM │ UTIL │ STATE) │
+│ ├─ 1247 │ GR │ 0x3 │ 2.1G │ 512M │ 82% │ RUNNING │
+│ ├─ 1247 │ CE0 │ 0x3 │ 128M │ 64M │ 41% │ RUNNING │
+│ ├─ 1247 │ DEC │ 0x3 │ 4.0G │ 256M │ 100% │ RUNNING │
+│ ├─ 1247 │ ENC │ 0x3 │ 1.5G │ 128M │ 82% │ RUNNING │
+│ ├─ 3891 │ GR │ 0x7 │ 512M │ 128M │ 12% │ WAITING │
+│ └─ 3891 │ CE0 │ 0x7 │ 64M │ 32M │ 5% │ WAITING │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ POWER / CLOCKS / THERMALS │
+│ ├─ Power Draw: 320W / 450W (71%) │ GPU Clock: 2850 MHz (max) │
+│ ├─ Mem Clock: 1313 MHz (max) │ SM Clock: 2850 MHz │
+│ ├─ Temp: 78°C (slowdown: 83°C) │ Fan: 65% (auto) │
+│ └─ PState: P0 (Performance) │ Perf Cap: None │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ GSP-RM TELEMETRY │
+│ ├─ RPC Queue: GR=12 CE=3 DEC=0 ENC=1 (depth / max=64) │
+│ ├─ Doorbell Rate: 45K/s │ Firmware Threads: 8/16 active │
+│ ├─ WPR Regions: 3 active │ Migration: IDLE │
+│ └─ Errors: 0 (CRC) 0 (Timeout) 0 (Page Fault) │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ PCIe / NVLink │
+│ ├─ Link: Gen4 x16 │ Throughput: 12.5 GB/s / 31.5 GB/s (40%) │
+│ ├─ Replay Timer: 0 │ NAK: 0 │ L0→L1: 12 L1→L0: 12 │
+│ └─ NVLink: N/A (single GPU) │
+└─────────────────────────────────────────────────────────────────────────────┘
+ [F1] Help [F2] GPU Select [F3] Engine Detail [F4] Memory Detail [F5] Processes [F6] GSP [F10] Quit
+```
+
+## Key Bindings
+| Key | Action |
+|-----|--------|
+| `↑/↓` | Navigate process list |
+| `←/→` | Switch GPU (multi-GPU) |
+| `F1` | Help |
+| `F2` | GPU Select menu |
+| `F3` | Engine detail view |
+| `F4` | Memory detail view |
+| `F5` | Process list (expanded) |
+| `F6` | GSP-RM telemetry |
+| `F7` | PCIe/NVLink detail |
+| `F8` | Power/thermal graphs |
+| `F9` | Configuration |
+| `F10` / `q` | Quit |
+| `Space` | Pause/resume updates |
+| `s` | Change sample interval (100ms-10s) |
+| `u` | Sort by GPU util / Memory / PID |
+| `k` | Kill process (send SIGTERM to guest) |
+| `e` | Export snapshot (JSON) |
+
+## Engine Detail View (F3)
+```
+┌─ GR Engine Detail ────────────────────────────────────────────────────────┐
+│ Channel 0 (PID 1247, CTX 0x3) │ Util: 82% │ Submitted: 1.2M ops │
+│ ├─ Pushbuffer: 0x7f8a... │ Fence: 0x123456 │ Pending: 47 │
+│ ├─ Context: 0x3 │ Page Tables: 0x100000 │ VRAM: 2.1G │
+│ └─ Perf: SM=82% TEX=45% L2=67% ROPS=34% │
+│ Channel 3 (PID 1247, CTX 0x3) │ Util: 12% │ Submitted: 45K ops │
+│ └─ ... │
+│ Channel 7 (PID 3891, CTX 0x7) │ Util: 5% │ Submitted: 12K ops │
+│ └─ ... │
+│ Channel 12 (PID 3891, CTX 0x7) │ Util: 0% │ Submitted: 0 ops │
+│ └─ IDLE │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Memory Detail View (F4)
+```
+┌─ Memory Detail ────────────────────────────────────────────────────────────┐
+│ VRAM Allocations (by PID) │
+│ ├─ 1247: 18.2G ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │
+│ ├─ 3891: 0.6G ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% │
+│ └─ System: 0.1G ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │
+│ │
+│ Fragmentation: 12% │ Largest Free Block: 2.1G │ Allocation Count: 347 │
+│ │
+│ BAR1 Mappings (CPU→GPU) │
+│ ├─ 0x7f8a00000000 → GPU VA 0x100000 (256M) RW PID 1247 │
+│ └─ 0x7f8b00000000 → GPU VA 0x200000 (128M) RO PID 3891 │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## GSP-RM Telemetry View (F6)
+```
+┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐
+│ RPC Queues (depth / max=64) │
+│ ├─ GR: ████████████░░░░░░░░░░ 12/64 │ Avg Latency: 2.3μs │
+│ ├─ CE0: ███░░░░░░░░░░░░░░░░░░░░░ 3/64 │ Avg Latency: 1.1μs │
+│ ├─ CE1: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │
+│ ├─ NVDEC: ██████████████████████ 64/64 │ Avg Latency: 4.7μs ⚠ SATURATED│
+│ ├─ NVENC: ████████████░░░░░░░░░░░░░ 1/64 │ Avg Latency: 3.2μs │
+│ └─ OFA: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │
+│ │
+│ Doorbell Statistics │
+│ ├─ Total Rate: 45,231/s │ Peak: 120,451/s │
+│ ├─ GR: 28,123/s │ CE: 8,456/s │ DEC: 5,234/s │ ENC: 3,418/s │
+│ └─ Errors: 0 (spurious) 0 (invalid) │
+│ │
+│ Firmware Threads (16 total) │
+│ ├─ Active: 8 │ Idle: 6 │ Blocked: 2 │
+│ ├─ Thread 0: GR Scheduler │ CPU: 45% │ RPC: 12,341/s │
+│ ├─ Thread 1: CE Manager │ CPU: 12% │ RPC: 3,456/s │
+│ ├─ Thread 2: DEC Manager │ CPU: 78% │ RPC: 5,234/s ⚠ HIGH │
+│ └─ ... │
+│ │
+│ WPR State │
+│ ├─ Region 0: 0x100000-0x200000 (1M) READ-ONLY PID 1247 │
+│ ├─ Region 1: 0x300000-0x400000 (1M) NO-ACCESS PID 1247 │
+│ └─ Region 2: 0x500000-0x600000 (1M) READ-WRITE PID 3891 │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Multi-GPU Support
+```
+rmtop -g 0 # Start on GPU 0
+rmtop -g all # Cycle through all GPUs (tab)
+rmtop --layout=grid # Show all GPUs in grid (requires wide terminal)
+```
+
+## Configuration (`~/.config/rmtop/config.toml`)
+```toml
+[general]
+sample_interval_ms = 500
+theme = "dark" # dark, light, mono
+show_gsp = true
+show_pcie = true
+show_nvlink = false
+
+[colors]
+engine_high = "red"
+engine_medium = "yellow"
+engine_low = "green"
+warning = "red"
+critical = "bright_red"
+
+[columns]
+# Show/hide columns in process list
+pid = true
+name = true
+engine = true
+context = true
+vram = true
+sysmem = true
+util = true
+state = true
+
+[alerts]
+gsp_queue_warning = 48 # 75% of max
+gsp_queue_critical = 56 # 87.5% of max
+temp_warning = 80
+temp_critical = 85
+power_warning = 400
+```
+
+## Build
+```bash
+cd open-iov/tools/rmtop
+make
+# Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess
+```
+
+## Architecture
+```
+rmtop/
+├── src/
+│ ├── main.c # ncurses init, main loop
+│ ├── ui.c # Layout, rendering, input handling
+│ ├── gpu_panel.c # GPU summary panel
+│ ├── engine_panel.c # Engine utilization bars
+│ ├── memory_panel.c # Memory breakdown
+│ ├── process_panel.c # Process/channel list
+│ ├── power_panel.c # Power/clocks/thermals
+│ ├── gsp_panel.c # GSP-RM telemetry
+│ ├── pcie_panel.c # PCIe/NVLink
+│ ├── detail_views.c # F3-F8 detail panels
+│ ├── sampler.c # Background sampling thread
+│ └── config.c # Config file parsing
+├── include/
+│ └── rmtop.h
+├── lib/
+│ ├── librmapi.a
+│ └── libgpumon.a
+└── Makefile
+```
+
+## Future Extensions
+- **Remote Dashboard**: WebSocket server for browser-based monitoring
+- **Historical Graphs**: In-terminal sparklines for utilization history
+- **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO
+- **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads
+- **Policy Visualization**: Show resserve reservations, quotas
+- **Migration Tracking**: Live migration progress bars
+- **Telemetry Export**: Background Prometheus/OpenTelemetry push \ No newline at end of file