From 2302fc09080bad51c73f293294e19fd4db464eb0 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- wiki/BAR-Emulation-Specification.md | 321 ++++++++++++++++++++++++++++++++++++ 1 file changed, 321 insertions(+) create mode 100644 wiki/BAR-Emulation-Specification.md (limited to 'wiki/BAR-Emulation-Specification.md') diff --git a/wiki/BAR-Emulation-Specification.md b/wiki/BAR-Emulation-Specification.md new file mode 100644 index 0000000..b8a46c3 --- /dev/null +++ b/wiki/BAR-Emulation-Specification.md @@ -0,0 +1,321 @@ +# BAR Emulation Specification (GSP 570.144W) + +## Overview +The GSP firmware implements BAR (Base Address Register) emulation for virtual GPU (vGPU) through the **BAR_TRAP** RPC (type 0x0A). The main handler at **0x04CB8188** (40 KB) manages BAR1 aperture, forced access, and memory backing. + +## Key Addresses + +| Component | Address | Description | +|-----------|---------|-------------| +| BAR Trap Handler | 0x04CB8188 | Main emulation logic (40 KB) | +| RPC Jump Table Entry | 0x057675B7 | BAR_TRAP in secondary dispatcher | +| String: BAR1 | 0x0406570E, 0x04085DE0 | BAR1 identifier | +| String: Bar1Rest | 0x040656DA | BAR1 restore state | +| String: ForceBarAccess | 0x040656F2 | Forced access control | +| String: Bar1Aperture | 0x04065722 | Aperture configuration | +| Memory Allocator | 0x429B450 | Arena for BAR backing store | +| Cache Management | 0x429B428 | Flush/invalidate | + +## Handler State Machine + +### Main Loop (0x04CB8188) +The handler runs a continuous event loop processing BAR trap requests: + +```c +while (true) { + // Wait for trap event (bitmask in unaff_s5) + event = wait_for_bar_event(); + + switch (event & 0x3F) { + case 0x01: // BAR1 write access + handle_bar1_write(); + break; + case 0x02: // BAR1 read access + handle_bar1_read(); + break; + case 0x08: // Aperture configuration + handle_aperture_config(); + break; + case 0x10: // ForceBarAccess + handle_force_access(); + break; + case 0x20: // Bar1Rest (restore) + handle_bar1_restore(); + break; + default: + return ERROR_UNKNOWN_EVENT; + } +} +``` + +### Event Bitmask +| Bit | Event | Description | +|-----|-------|-------------| +| 0 | BAR1_WRITE | Guest VF wrote to BAR1 | +| 1 | BAR1_READ | Guest VF read from BAR1 | +| 3 | APERTURE | Aperture size/offset change | +| 4 | FORCE_ACCESS | ForceBarAccess triggered | +| 5 | BAR1_REST | Restore BAR1 state | + +## Memory Management + +### Allocator Arena (0x429B450) +```c +// Allocate backing store for BAR emulation +void *bar_alloc(size_t size) { + // Calls UNK_ram_058A1A9C(0x429B450, size) + // Returns pointer in 0x429B000 region +} + +// Free backing store +void bar_free(void *ptr) { + // Calls UNK_ram_05895BF4(0x429B428, ptr) +} +``` + +### Allocation Sizes +- **Standard BAR1 page**: 0x1000 (4 KB) +- **Aperture buffer**: 0x40000 (256 KB) max +- **Force access buffer**: 0x1000 (4 KB) + +### Cache Coherency +```c +// Flush cache for DMA +void bar_cache_flush(void *addr, size_t size) { + UNK_ram_05895BF4(0x429B428, addr); + UNK_ram_05895BF4(0x429B428, addr + size); +} +``` + +## BAR1 Emulation Logic + +### Write Path (Event 0x01) +```c +int handle_bar1_write(bar_trap_t *trap) { + // 1. Validate offset within aperture + if (trap->offset >= bar1_aperture_size) + return ERROR_OVERFLOW; + + // 2. Check ForceBarAccess flag + if (force_bar_access && !(trap->flags & FORCE_ALLOWED)) + return ERROR_PERMISSION; + + // 3. Translate to backing store + uint8_t *backing = bar1_backing + trap->offset; + + // 4. Perform write (1/2/4/8 bytes) + switch (trap->size) { + case 1: *backing = trap->value; break; + case 2: *(uint16_t*)backing = trap->value; break; + case 4: *(uint32_t*)backing = trap->value; break; + case 8: *(uint64_t*)backing = trap->value; break; + } + + // 5. Flush to device if needed + if (device_needs_flush) + bar_cache_flush(backing, trap->size); + + return OK; +} +``` + +### Read Path (Event 0x02) +```c +int handle_bar1_read(bar_trap_t *trap) { + if (trap->offset >= bar1_aperture_size) + return ERROR_OVERFLOW; + + uint8_t *backing = bar1_backing + trap->offset; + + switch (trap->size) { + case 1: *trap->out_value = *backing; break; + case 2: *trap->out_value = *(uint16_t*)backing; break; + case 4: *trap->out_value = *(uint32_t*)backing; break; + case 8: *trap->out_value = *(uint64_t*)backing; break; + } + + return OK; +} +``` + +### Aperture Configuration (Event 0x08) +```c +int handle_aperture_config(bar_config_t *cfg) { + // Validate new aperture + if (cfg->size > MAX_APERTURE_SIZE) // 0x40000 + return ERROR_INVALID; + + if (cfg->offset & 0xFFF) // Must be 4KB aligned + return ERROR_ALIGNMENT; + + // Reallocate backing store if size changed + if (cfg->size != bar1_aperture_size) { + void *new_backing = bar_alloc(cfg->size); + if (!new_backing) return ERROR_NOMEM; + + // Copy existing data + memcpy(new_backing, bar1_backing, + min(cfg->size, bar1_aperture_size)); + + bar_free(bar1_backing); + bar1_backing = new_backing; + bar1_aperture_size = cfg->size; + } + + bar1_aperture_offset = cfg->offset; + return OK; +} +``` + +### ForceBarAccess (Event 0x10) +```c +int handle_force_access(force_access_t *fa) { + // ForceBarAccess allows hypervisor to bypass VF permissions + // Used for live migration, checkpoint/restore + + force_bar_access = fa->enable; + force_access_vf = fa->vf_id; + force_access_range = fa->range; + + // Allocate shadow buffer for forced operations + if (fa->enable) { + force_buffer = bar_alloc(0x1000); + if (!force_buffer) return ERROR_NOMEM; + } else { + bar_free(force_buffer); + force_buffer = NULL; + } + + return OK; +} +``` + +### Bar1Rest - State Restore (Event 0x20) +```c +int handle_bar1_restore(restore_state_t *rs) { + // Restore BAR1 state from checkpoint + // Used during VF live migration + + bar1_aperture_offset = rs->aperture_offset; + bar1_aperture_size = rs->aperture_size; + force_bar_access = rs->force_access; + + // Restore backing store from serialized data + if (rs->backing_size > 0) { + void *new_backing = bar_alloc(rs->backing_size); + if (!new_backing) return ERROR_NOMEM; + + memcpy(new_backing, rs->backing_data, rs->backing_size); + bar_free(bar1_backing); + bar1_backing = new_backing; + } + + return OK; +} +``` + +## vGPU-IO Integration + +### Kernel BAR Trap Registration +```c +struct gsp_bar_ops { + int (*bar_trap)(struct gsp_device *gsp, + struct bar_trap_msg *msg); + int (*configure_aperture)(struct gsp_device *gsp, + uint32_t vf_id, + uint64_t offset, uint64_t size); + int (*force_access)(struct gsp_device *gsp, + uint32_t vf_id, bool enable); + int (*save_state)(struct gsp_device *gsp, + uint32_t vf_id, void *buf, size_t *size); + int (*restore_state)(struct gsp_device *gsp, + uint32_t vf_id, void *buf, size_t size); +}; + +static const struct gsp_bar_ops gsp_bar_ops = { + .bar_trap = gsp_bar_trap_handler, + .configure_aperture = gsp_bar_config_aperture, + .force_access = gsp_bar_force_access, + .save_state = gsp_bar_save_state, + .restore_state = gsp_bar_restore_state, +}; +``` + +### VF BAR Configuration +```c +int gsp_vf_bar_init(struct gsp_vf *vf) { + struct bar_config cfg = { + .vf_id = vf->id, + .bar = 1, // BAR1 + .aperture_offset = vf->bar1_offset, + .aperture_size = vf->bar1_size, + .flags = BAR_FLAG_COHERENT, + }; + + return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP, + BAR_SUBCMD_CONFIG, &cfg, sizeof(cfg)); +} +``` + +### Live Migration Support +```c +// Save BAR state for migration +int gsp_vf_bar_save(struct gsp_vf *vf, void *buf, size_t *size) { + struct bar_save_msg msg = { + .vf_id = vf->id, + .bar = 1, + }; + + return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP, + BAR_SUBCMD_SAVE, &msg, sizeof(msg), + buf, size); +} + +// Restore BAR state on destination +int gsp_vf_bar_restore(struct gsp_vf *vf, void *buf, size_t size) { + struct bar_restore_msg msg = { + .vf_id = vf->id, + .bar = 1, + .data = buf, + .data_size = size, + }; + + return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP, + BAR_SUBCMD_RESTORE, &msg, sizeof(msg)); +} +``` + +## Error Handling + +| Error | Code | Recovery | +|-------|------|----------| +| OOM (0x51) | Allocator exhausted | Shrink aperture, retry | +| Timeout (0x1E) | Cache flush timeout | Reset BAR, reinitialize | +| Invalid (0x1F) | Bad aperture config | Validate alignment/size | +| Permission (0xFFFF) | ForceAccess violation | Check VF permissions | +| Overflow (0x25) | Offset beyond aperture | Bounds check in VF driver | + +## Debugging + +### Key Registers to Monitor +``` +0x04296048 : Global error flag (cRam0000000004296048) +0x0429B750 : Error status (0x111300 offset) +0x429B428 : Cache management base +0x429B450 : Allocator arena base +``` + +### Log Strings +Search for these in GSP logs: +- `BAR1` - BAR1 access +- `Bar1Rest` - State restore +- `ForceBarAccess` - Forced access +- `Bar1Aperture` - Aperture changes +- `Bar1Size` - Size queries + +--- + +## Reference +- Handler: `FUN_ram_04cb8188_ram_04cb8188.c` (40 KB) +- RPC Type: 0x0A (BAR_TRAP) +- Jump Table Entry: 0x057675B7 \ No newline at end of file -- cgit v1.2.3