1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
# vgsp.ko integration (v2, live path)
The bridge is a userspace daemon with a fixed-size file protocol so the vGPU
kernel module drives it without sharing kernel/userspace memory. The v2 records
are frozen in `PROTOCOL.md` and mirrored by static asserts in
`kernel/gsp_cuda_bridge.c` and `tools/cuda-bridge/cuda_bridge.c`.
## Expected flow
1. **Detect launch.** `vgsp.ko`'s service thread decodes the guest's inline QMD
on the emulated GR compute channel (`gsp_sema_service.c`), classifies the
launch shape (mail vs vadd) and reads the parameter bank's pointers/count.
2. **Marshal + publish.** The module resolves the guest source VAs, copies the
inputs into its own buffer (vadd: `a` then `b`; mail: none) and publishes a
256-byte `CUBB2` job with a fresh `seq` in `cuda_bridge_job` while serving
the input blob from `cuda_bridge_in`. One job is published at a time
(`CUBB2_PEND_MAX` slots, oldest first).
3. **Daemon executes.** `cuda_bridge --bridge --job
/sys/kernel/debug/vgsp/cuda_bridge_job --in
/sys/kernel/debug/vgsp/cuda_bridge_in --out
/sys/kernel/debug/vgsp/cuda_bridge_out --done
/sys/kernel/debug/vgsp/cuda_bridge_done` polls the job file (default 50 ms),
reads the inputs, executes the requested kernel on the physical GPU
(embedded PTX; `cuMemcpyHtoD` / `cuLaunchKernel` / `cuMemcpyDtoH`), writes
the output blob to `cuda_bridge_out`, then always writes a `CBD2` done
record to `cuda_bridge_done`.
4. **Module completes the guest.** `vgsp.ko` pops done records on the service
thread, matches `seq`, copies the output blob through the VA translation
back into the guest destination, applies the QMD release semaphore, and the
guest's `cudaDeviceSynchronize` completes.
Everything is inert until a launch is seen; the path can be switched off with
the `cuda_bridge` module parameter (default on).
## Daemon invocation (frozen CLI)
```
cuda_bridge --bridge \
--job /sys/kernel/debug/vgsp/cuda_bridge_job \
--in /sys/kernel/debug/vgsp/cuda_bridge_in \
--out /sys/kernel/debug/vgsp/cuda_bridge_out \
--done /sys/kernel/debug/vgsp/cuda_bridge_done
```
`--poll-ms` overrides the 50 ms default; `--once` exits after one job
(bring-up only). The legacy v1 `--job J --done D --ram R` GPA mode is separate
and unchanged.
## Debugfs permissions
`cuda_bridge_job`/`_in` are `0444` and `_out`/`_done` are `0200`, owned by
root. When the daemon runs unprivileged, the runner opens them first:
```
for f in job in out done status; do
sudo chmod 666 /sys/kernel/debug/vgsp/cuda_bridge_$f 2>/dev/null
done
```
## seq semantics
* The module's counter starts at 1 and only increases; a job record is
immutable while pending.
* The daemon records `last_seq` and ignores any valid record with
`seq <= last_seq` (duplicate/replay protection). The first valid record
after daemon startup always executes, so a daemon restart can re-run at most
the single stale job still parked in the interface.
* A done record whose `seq` matches no pending module job is counted `stale`
and dropped; malformed magic/version is counted `malformed`. Both are
visible in `cuda_bridge_status`.
## Logging
The daemon logs one line per executed job (stdout only; correctness never
depends on it):
```
CUBB2 seq=42 op=2 n=1048576 gpu="NVIDIA GeForce RTX 2080 Super with Max-Q Design" \
kernel_ns=41919 status=ok checksum=392344090.500000 bad=0 out_bytes=4194304
```
`status=err` lines are preceded by a `bridge: job seq=... rejected/failed: ...`
stderr line naming the cause (bounds, bad opcode, input transport, output
transport). Every job — success or failure — gets a done record; failures
never leave the module waiting.
## Limits and error mapping
* `n <= 1<<26`; `in_bytes`, `out_bytes <= 32 MiB`; mail `n <= 4096` (module
param-length limit). Violations: `status = 1` (`bad job`), `err_code = 22`
(`EINVAL`).
* Bad opcode: `status = 1`, `err_code = opcode`.
* Input read failure: `status = 3`, `err_code = 5` (`EIO`); allocation
failure: `err_code = 12` (`ENOMEM`).
* CUDA failure: `status = 2`, `err_code = CUresult`.
* Output write failure: `status = 3`, `err_code = errno`.
* `bad` is data quality, not execution failure: `status` stays 0 when the
kernel ran and the output was transported.
## Risks / open questions
* **Single writer per job interface.** One job is visible at a time and the
module serialises publication; the daemon's `last_seq` only protects
duplicates. Multiple daemons against one mdev are unsupported.
* **Daemon lifetime / restarts.** On restart `last_seq` resets and the one
parked job may run again. The module can clear or bump the job when (re)starting
the daemon; the re-run is idempotent for mail/vadd.
* **Timeouts.** A stuck job has no protocol-level timeout; the module bounds
`n`/bytes and the runner has bounded waits. A watchdog belongs on the
module side.
* **Device selection.** The daemon always uses `cuDeviceGet(0)`; multi-GPU
hosts need a `--device` selector and the module must agree on the physical
GPU behind the mdev.
* **PTX target.** The embedded PTX is `sm_75` (JIT-ed by the driver for the
physical GPU); a newer-only GPU needs the build-time `PTX_ARCH` bumped.
|
