# 30 — GPGPU-Sim integration (in development)

> **Status: in development (Phase 0 scaffold).** This page describes a
> development-only overlay that is being prepared to give the debugger an
> off-hardware GPU execution substrate. It makes **no accuracy, timing,
> performance or support claim**, is **not calibrated to silicon**, and is
> not a product or guest component. Nothing here authorises GPU hardware,
> kernel-module, mdev or guest access. Thresholds for the capabilities below
> exist only as unfiled drafts in
> [`../gpgpu-sim/ADMISSION.md`](../gpgpu-sim/ADMISSION.md).

## 1. Purpose

QEMU (with the `vgsp` host module) and the native reference core under `sim/`
cover the debugger's instruction-level needs today. The next step is an
execution substrate that can host, on one deterministic clock, both a
GPU microarchitecture model and the GSP/microcontroller models the program
needs, while reusing the debugger's session format, hooks/watchpoints, Ghidra
bridge and MCP surface.

GPGPU-Sim is the chosen base: it is open source (BSD-3-Clause, University of
British Columbia), data-driven, and already models Turing SM75 in its tested
configurations. The overlay in
[`../gpgpu-sim/`](../gpgpu-sim/README.md) tracks the project's changes to
upstream without forking its history: a pinned clone plus an ordered patch
set.

## 2. Status (Phase 0)

What exists today:

| Item | State |
|---|---|
| Overlay layout + pinned upstream policy | done — [`../gpgpu-sim/UPSTREAM.md`](../gpgpu-sim/UPSTREAM.md) (pin `90ec339`, tag v4.0.1) |
| Idempotent setup/build script | done — `../gpgpu-sim/setup.sh` (clone/checkout, ordered patches, CUDA-`CUDA_INSTALL_PATH` build; `--no-build` supported) |
| Turing TU104 (RTX 2080 SUPER) config scaffold | done — `../gpgpu-sim/configs/turing-tu104-2080super/` |
| Config consistency test | done — `../gpgpu-sim/scripts/config_sanity.py`, wired into `make test` |
| Run wrapper for CUDA binaries | done — `../gpgpu-sim/scripts/run_sim.sh` (LD_LIBRARY_PATH substitution per upstream) |
| Admission records | drafts only — [`../gpgpu-sim/ADMISSION.md`](../gpgpu-sim/ADMISSION.md), pending maintainer filing |
| CUDA 12.x build/runtime port | **done (WP0.1)** — patches `0001`/`0002`; `../gpgpu-sim/BUILD-PORT.md` records the pinned CUDA 12.0 / gcc 13 build and `vectorAdd` functional+perf green |
| Calibrated TU104 model | **not done** (Phase 1) |
| GSP core + Turing MCU models | **not done** (Phase 1.5, behind device-map and model gates) |

## 3. What the TU104 scaffold encodes, and how

The config is derived from the upstream SM75_RTX2060 Turing config, with only
the device-level parameters public specifications distinguish. Values carried
from public sources:

* 48 SMs (3072 CUDA cores at 64 FP32/SM) → `-gpgpu_n_clusters 48`;
* 8 GB GDDR6, 256-bit, 15.5 Gbps effective → recorded as machine-checkable
  `# SPEC:` markers plus the DRAM model (burst 16, QDR ratio 4, DRAM command
  clock 3875 MHz); **memory capacity itself is not modelled by GPGPU-Sim 4.x**
  and is documented, not asserted;
* ~496 GB/s → the modeled module count/geometry is chosen so
  `buswidth × ratio × DRAM clock × modules` equals 496 GB/s, and
  `config_sanity.py` recomputes it;
* 4 MB L2 → cache geometry × modules × sub-partitions equals 4 MiB, also
  recomputed by the test;
* 64 ROPs / 192 TMUs → recorded as `# SPEC:` markers (GPGPU-Sim 4.x has no
  count knobs for these);
* base ~1650 MHz / boost ~1815 MHz → base clock active, boost mode commented;
* compute capability 7.5, PCIe 3.0 x16 → CC asserted; PCIe is not modelled.

Anything not tied to a published number keeps an explicit `# UNKNOWN:`
marker (latencies, L1/L2/DRAM timing, address mapping, Booksim topology
size). See the config header for the full list.

## 4. How to build and run

```bash
cd open-iov/debug/risc-v-gsp-rm-vm/gpgpu-sim

./setup.sh                    # clone pin + apply patches + build
python3 scripts/config_sanity.py
./scripts/run_sim.sh --help
./scripts/run_sim.sh ~/path/to/vectorAdd
```

`setup.sh` is idempotent, never uses `sudo`, and caps compile jobs at 4 by
default (`JOBS` overrides). The CUDA toolkit is selected with
`CUDA_INSTALL_PATH` (default `/usr/local/cuda`). On hosts whose CUDA is not
yet ported, the build step stops with a clear error; `--no-build` still
materialises the checkout and patches.

## 5. What is NOT claimed

* No accuracy, cycle, latency, bandwidth or correlation claim: the config is
  a documented scaffold, and config-only modeling is treated as a failure
  mode, not a result.
* No calibrated model, no GSP/MCU/fabric/guest simulation yet; the real GSP
  image is not loaded by this overlay.
* No support or product claim of any kind; development-only tooling, never
  shipped, never in a guest.
* No hardware access from any script in this overlay.

## 6. Admission drafts

The five draft records (build/run, TU104 calibration, GSP-in-sim, MCU
cross-controller, sim debugger backend) with scenario, measurable outcome,
baseline, threshold and failure signal live in
[`../gpgpu-sim/ADMISSION.md`](../gpgpu-sim/ADMISSION.md). They are drafts and
require maintainer filing under the project admission rule; until then these
capabilities are pilots at most.