VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl handlers

- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create()
- Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions
- Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP
- Added config space read/write handlers at offset 0x70000000000
- Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers)
- Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL)
- Added udev rules for vfio group permissions
- Created stack_manager.sh unified tool
- Fixed mdev remove crash with deferred cleanup in vfio_device_release callback
- Kernel 6.8+ IOMMUFD compatibility
- QEMU launches successfully with vGPU passthrough

1 files changed, 227 insertions, 0 deletions

diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md
new file mode 100644
index 0000000..6408268
--- /dev/null
+++ b/docs/tools/rmtop.md
@@ -0,0 +1,227 @@
+# rmtop - GPU htop Equivalent
+
+## Overview
+`rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry.
+
+## Source Code Reuse
+Same recycled components as `rmtool`:
+- `gpumon_perfmon` - sampling infrastructure, perfmon counters
+- `nvidia-smi` - GPU enumeration, NVML-style queries
+- `nv2080` - direct RM control for real-time state
+- `resserve` - resource server state
+- `resman` - resource manager heaps/clients
+
+## UI Layout
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ rmtop 1.0  │  GPU 0: RTX 5090 (GB202)  │  Driver: 610.XX  │  14:32:15      │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ ENGINES          │ UTIL  │  FREQ   │  POWER  │  TEMP  │  ACTIVE CHANNELS   │
+│ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │  320W   │  78°C  │  ch: 0,3,7,12   │
+│ ├─ CE0 (Copy)    │ ████░░░░░░ 41% │ 2100MHz │         │        │  ch: 1,4        │
+│ ├─ CE1 (Copy)    │ ░░░░░░░░░░  0% │  300MHz │         │        │  -              │
+│ ├─ NVDEC0        │ ██████████ 100% │ 1800MHz │         │        │  ch: 2          │
+│ ├─ NVDEC1        │ ░░░░░░░░░░  0% │  300MHz │         │        │  -              │
+│ ├─ NVENC0        │ ████████░░ 82% │ 1800MHz │         │        │  ch: 5          │
+│ ├─ NVENC1        │ ░░░░░░░░░░  0% │  300MHz │         │        │  -              │
+│ ├─ NVJPG         │ ░░░░░░░░░░  0% │  300MHz │         │        │  -              │
+│ └─ OFA           │ ░░░░░░░░░░  0% │  300MHz │         │        │  -              │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ MEMORY           │ USED  │  TOTAL  │  FREE   │  FRAG  │  BAR1              │
+│ ├─ VRAM          │ 18.2G │  24.0G  │  5.8G   │  12%   │  256M / 256M     │
+│ ├─ System RAM    │  4.1G │  64.0G  │ 59.9G   │   3%   │                    │
+│ └─ BAR1 (CPU)    │  12M  │  256M   │ 244M    │   1%   │                    │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ CHANNELS / CONTEXTS (PID │ ENG │ CTX │  VRAM  │  SYSMEM  │  UTIL  │ STATE) │
+│ ├─  1247  │ GR  │  0x3  │  2.1G  │  512M   │  82%   │  RUNNING         │
+│ ├─  1247  │ CE0 │  0x3  │  128M  │  64M    │  41%   │  RUNNING         │
+│ ├─  1247  │ DEC │  0x3  │  4.0G  │  256M   │ 100%   │  RUNNING         │
+│ ├─  1247  │ ENC │  0x3  │  1.5G  │  128M   │  82%   │  RUNNING         │
+│ ├─  3891  │ GR  │  0x7  │  512M  │  128M   │  12%   │  WAITING         │
+│ └─  3891  │ CE0 │  0x7  │  64M   │  32M    │   5%   │  WAITING         │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ POWER / CLOCKS / THERMALS                                                  │
+│ ├─ Power Draw: 320W / 450W (71%)  │  GPU Clock: 2850 MHz (max)           │
+│ ├─ Mem Clock: 1313 MHz (max)      │  SM Clock: 2850 MHz                  │
+│ ├─ Temp: 78°C (slowdown: 83°C)    │  Fan: 65% (auto)                     │
+│ └─ PState: P0 (Performance)       │  Perf Cap: None                      │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ GSP-RM TELEMETRY                                                           │
+│ ├─ RPC Queue: GR=12  CE=3  DEC=0  ENC=1  (depth / max=64)                │
+│ ├─ Doorbell Rate: 45K/s  │  Firmware Threads: 8/16 active                │
+│ ├─ WPR Regions: 3 active  │  Migration: IDLE                             │
+│ └─ Errors: 0 (CRC)  0 (Timeout)  0 (Page Fault)                          │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ PCIe / NVLink                                                              │
+│ ├─ Link: Gen4 x16  │  Throughput: 12.5 GB/s / 31.5 GB/s (40%)           │
+│ ├─ Replay Timer: 0  │  NAK: 0  │  L0→L1: 12  L1→L0: 12                   │
+│ └─ NVLink: N/A (single GPU)                                                │
+└─────────────────────────────────────────────────────────────────────────────┘
+  [F1] Help  [F2] GPU Select  [F3] Engine Detail  [F4] Memory Detail  [F5] Processes  [F6] GSP  [F10] Quit
+```
+
+## Key Bindings
+| Key | Action |
+|-----|--------|
+| `↑/↓` | Navigate process list |
+| `←/→` | Switch GPU (multi-GPU) |
+| `F1` | Help |
+| `F2` | GPU Select menu |
+| `F3` | Engine detail view |
+| `F4` | Memory detail view |
+| `F5` | Process list (expanded) |
+| `F6` | GSP-RM telemetry |
+| `F7` | PCIe/NVLink detail |
+| `F8` | Power/thermal graphs |
+| `F9` | Configuration |
+| `F10` / `q` | Quit |
+| `Space` | Pause/resume updates |
+| `s` | Change sample interval (100ms-10s) |
+| `u` | Sort by GPU util / Memory / PID |
+| `k` | Kill process (send SIGTERM to guest) |
+| `e` | Export snapshot (JSON) |
+
+## Engine Detail View (F3)
+```
+┌─ GR Engine Detail ────────────────────────────────────────────────────────┐
+│ Channel 0 (PID 1247, CTX 0x3)  │  Util: 82%  │  Submitted: 1.2M ops     │
+│   ├─ Pushbuffer: 0x7f8a...    │  Fence: 0x123456  │  Pending: 47         │
+│   ├─ Context: 0x3  │  Page Tables: 0x100000  │  VRAM: 2.1G              │
+│   └─ Perf: SM=82%  TEX=45%  L2=67%  ROPS=34%                             │
+│ Channel 3 (PID 1247, CTX 0x3)  │  Util: 12%  │  Submitted: 45K ops      │
+│   └─ ...                                                                        │
+│ Channel 7 (PID 3891, CTX 0x7)  │  Util: 5%   │  Submitted: 12K ops      │
+│   └─ ...                                                                        │
+│ Channel 12 (PID 3891, CTX 0x7) │  Util: 0%   │  Submitted: 0 ops        │
+│   └─ IDLE                                                                       │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Memory Detail View (F4)
+```
+┌─ Memory Detail ────────────────────────────────────────────────────────────┐
+│ VRAM Allocations (by PID)                                                   │
+│ ├─ 1247: 18.2G  ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │
+│ ├─ 3891:  0.6G  ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  2% │
+│ └─ System: 0.1G  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  0% │
+│                                                                              │
+│ Fragmentation: 12%  │  Largest Free Block: 2.1G  │  Allocation Count: 347 │
+│                                                                              │
+│ BAR1 Mappings (CPU→GPU)                                                     │
+│ ├─ 0x7f8a00000000 → GPU VA 0x100000  (256M)  RW  PID 1247                  │
+│ └─ 0x7f8b00000000 → GPU VA 0x200000  (128M)  RO  PID 3891                  │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## GSP-RM Telemetry View (F6)
+```
+┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐
+│ RPC Queues (depth / max=64)                                                │
+│ ├─ GR:     ████████████░░░░░░░░░░  12/64  │  Avg Latency: 2.3μs           │
+│ ├─ CE0:    ███░░░░░░░░░░░░░░░░░░░░░  3/64   │  Avg Latency: 1.1μs           │
+│ ├─ CE1:    ░░░░░░░░░░░░░░░░░░░░░░░░  0/64   │  -                            │
+│ ├─ NVDEC:  ██████████████████████  64/64  │  Avg Latency: 4.7μs  ⚠ SATURATED│
+│ ├─ NVENC:  ████████████░░░░░░░░░░░░░  1/64   │  Avg Latency: 3.2μs           │
+│ └─ OFA:    ░░░░░░░░░░░░░░░░░░░░░░░░  0/64   │  -                            │
+│                                                                              │
+│ Doorbell Statistics                                                        │
+│ ├─ Total Rate: 45,231/s  │  Peak: 120,451/s                              │
+│ ├─ GR: 28,123/s  │  CE: 8,456/s  │  DEC: 5,234/s  │  ENC: 3,418/s        │
+│ └─ Errors: 0 (spurious)  0 (invalid)                                       │
+│                                                                              │
+│ Firmware Threads (16 total)                                                │
+│ ├─ Active: 8  │  Idle: 6  │  Blocked: 2                                    │
+│ ├─ Thread 0: GR Scheduler      │  CPU: 45%  │  RPC: 12,341/s              │
+│ ├─ Thread 1: CE Manager        │  CPU: 12%  │  RPC: 3,456/s               │
+│ ├─ Thread 2: DEC Manager       │  CPU: 78%  │  RPC: 5,234/s  ⚠ HIGH       │
+│ └─ ...                                                                        │
+│                                                                              │
+│ WPR State                                                                   │
+│ ├─ Region 0: 0x100000-0x200000  (1M)  READ-ONLY  PID 1247                 │
+│ ├─ Region 1: 0x300000-0x400000  (1M)  NO-ACCESS  PID 1247                 │
+│ └─ Region 2: 0x500000-0x600000  (1M)  READ-WRITE PID 3891                 │
+└────────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Multi-GPU Support
+```
+rmtop -g 0          # Start on GPU 0
+rmtop -g all        # Cycle through all GPUs (tab)
+rmtop --layout=grid # Show all GPUs in grid (requires wide terminal)
+```
+
+## Configuration (`~/.config/rmtop/config.toml`)
+```toml
+[general]
+sample_interval_ms = 500
+theme = "dark"          # dark, light, mono
+show_gsp = true
+show_pcie = true
+show_nvlink = false
+
+[colors]
+engine_high = "red"
+engine_medium = "yellow"
+engine_low = "green"
+warning = "red"
+critical = "bright_red"
+
+[columns]
+# Show/hide columns in process list
+pid = true
+name = true
+engine = true
+context = true
+vram = true
+sysmem = true
+util = true
+state = true
+
+[alerts]
+gsp_queue_warning = 48    # 75% of max
+gsp_queue_critical = 56   # 87.5% of max
+temp_warning = 80
+temp_critical = 85
+power_warning = 400
+```
+
+## Build
+```bash
+cd open-iov/tools/rmtop
+make
+# Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess
+```
+
+## Architecture
+```
+rmtop/
+├── src/
+│   ├── main.c              # ncurses init, main loop
+│   ├── ui.c                # Layout, rendering, input handling
+│   ├── gpu_panel.c         # GPU summary panel
+│   ├── engine_panel.c      # Engine utilization bars
+│   ├── memory_panel.c      # Memory breakdown
+│   ├── process_panel.c     # Process/channel list
+│   ├── power_panel.c       # Power/clocks/thermals
+│   ├── gsp_panel.c         # GSP-RM telemetry
+│   ├── pcie_panel.c        # PCIe/NVLink
+│   ├── detail_views.c      # F3-F8 detail panels
+│   ├── sampler.c           # Background sampling thread
+│   └── config.c            # Config file parsing
+├── include/
+│   └── rmtop.h
+├── lib/
+│   ├── librmapi.a
+│   └── libgpumon.a
+└── Makefile
+```
+
+## Future Extensions
+- **Remote Dashboard**: WebSocket server for browser-based monitoring
+- **Historical Graphs**: In-terminal sparklines for utilization history
+- **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO
+- **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads
+- **Policy Visualization**: Show resserve reservations, quotas
+- **Migration Tracking**: Live migration progress bars
+- **Telemetry Export**: Background Prometheus/OpenTelemetry push
\ No newline at end of file