summaryrefslogtreecommitdiffstats
path: root/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md
blob: 48e9f35b20e154612c2d31302bd245c1a9d7321a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
# ANALYSIS - EGL 3D-engine selection wall (570.144 guest userspace)

Status: OPEN (2026-10-23). The EGL-device path initialises (devices enumerate,
`eglInitialize` succeeds, `vendor=NVIDIA`), but `eglMakeCurrent` fails with
`EGL_BAD_ALLOC (0x3003)` because the per-device EGL-core client creator
returns NULL after a lower-layer engine-init call fails.

This document records the *shipped-binary* evidence only (disassembly +
live gdb traces). Method: run the probe under gdb with ASLR disabled
(gdb's default) so library offsets are stable, set breakpoints at the
recorded offsets, and read the arguments/returns.

## 1. The failing chain (libnvidia-eglcore.so.570.144)

```
eglMakeCurrent
  -> NvGlEglApiInit family: per-device EGL-core client creator  (+0xa246f0)
       -> +0xa06ed0  (device/engine selection wrapper)
            -> +0x9fe790  (engine-list walk)
                 list head: *( *(device + 0x28738) + 8 )
                 filters per engine:  engine[0x59] & 0x40
                                      engine[0x70] == 0
                 per-engine callback: *0x70( *(*(engine+0x28738)+0x10) )
                      -> +0xa22000  (callback)
                           rbx = *(reqs + 0xa0)          (device object)
                           mode = *(reqs + 0x20)
                           mode 6 -> +0xa09830
                           mode 7 -> +0xa0a4d0
                           else   -> +0x9fdfd0(device, engine, obj, reqs)
       NULL -> EGL_BAD_ALLOC
```

`+0x9fdfd0` ends in the rejection site:

```
9fe146: rax = *(r14 + 0x10)        ; r14 = *(engine + 0x28738)
9fe14d: rdi = *(rbx + 0x20)        ; rbx = *(reqs + 0xa0)  (device object)
9fe151: call *0x30(rax)            ; vtable slot 0x30
9fe154: test eax,eax
9fe156: je   9fe170                ; 0 == continue (success)
9fe158: free(tmp); return 0        ; non-zero == reject
```

## 2. The rejecting method (libnvidia-glsi.so.570.144)

The vtable at `*(*(engine+0x28738)+0x10)` is a *glsi* device-engine vtable;
its slot 0x30 resolves (live gdb: vtable `0x7ffff71ddfe0`, method
`+0xa29440` in eglcore terms = glsi `+0x5f480` base-adjusted) to the glsi
engine-init entry:

```
glsi +0x5f480(device, reqs):
  5f4c0: fd  = *(*(*(dev+0x108)+0x58))     ; int fd (nvidia device)
  5f4c8: rsi = *(dev+0x118)
  5f4dc: call +0x1aba0(fd, rsi, reqs)      ; FIRST call
  5f52e: call +0x1af20(...)                ; second call
```

Live trace (probe `egl_frame` under gdb, breakpoint at the callback's call
site): the **first** glsi call returns `0xee00001`; the caller propagates it
and the engine is rejected. `0xee00001` is a glsi-internal status produced by
two sites (`glsi +0x31f84`, `+0x33910`) that set flag bit `0x20` in the
device's `+0xb84` word and return "busy"-style; the engine-init state machine
(`glsi +0x301b0` and callees `+0x2fbe0` -> `+0x29760` -> `+0x32110`/`+0x2dc70`)
is what fails.

## 3. What is ruled out

* **Not an RM-control error.** During the failing probe the emulated GSP
  answers every `GSP_RM_CONTROL` with `status=0x0` (including the
  `0x20810107` BINAPI query that was previously suspected; that one is a red
  herring - a real class-0x2081 object on the live session answers
  `NV_OK`/`{0,0}` exactly like the emulation).
* **Not the engine-list shape alone.** The single candidate engine passes the
  walk's own filters (`flag59 & 0x40` set, `f70 == 0`), so the list contains
  a usable engine; the failure is inside the per-engine init.
* **Not ASLR-dependent.** All offsets above are stable under gdb
  (`set disable-randomization on`).

## 4. Entry-trace update (2026-10-23, r543 boot)

A batch-gdb entry trace (breakpoints on the glsi engine-init entry and its
inner functions, breakpoint set after `eglInitialize` so the vendor libraries
are mapped; ASLR disabled reproduces glsi base `0x7ffff7e2a000`) captured the
failing call order:

```
glsi+0x5f480 (engine-init)
  glsi+0x1aba0 (first inner call)
    glsi+0x22f30   device lookup by GPU id   (miss -> 0xee00002)
    glsi+0x2c360   object lookup             (miss -> 0xee00003)
    glsi+0x33b80   device-flags + state      <-- the failing layer
      glsi+0x301b0 (state machine, entered twice)
        glsi+0x3a6f0  (entered ~7x)
        glsi+0x2ac10  (entered twice)
  -> EGL_CURRENT_FAIL err=0x3003 (eglcore rejects the engine)
```

`+0x5f480` returns as soon as `+0x1aba0` reports non-zero; `+0x1aba0`
propagates the first failure of `+0x22f30` / `+0x2c360` / `+0x33b80`. In the
trace all three were entered, and the failure lands inside `+0x33b80`'s
state-machine call. `+0x33b80` reads the device flags word at `+0xb84`; on a
state-machine failure with the retry bit set it marks the device busy
(`or 0x20` into the state record at `device + idx*0xdc8 + 0x291b4`) and
returns `0xee00001` once (`glsi+0x30ef2` inside its callee `+0x30b20`);
the higher layer reads `+0xb84` bit 0x20 and rejects the engine for this
attempt without retry. `+0x22f30` is called ~1300 times during the probe
(per-device/per-engine enumeration) - the failure is not a lookup miss.

The state machine (`+0x301b0`) consumes per-engine context records; its
inputs are the fields the caller passes in (`+0x95` mode byte, `+0x38`
record array, `+0x74` id, `+0x98` index) plus the device flags. The next
step is a return-value trace of `+0x3a6f0`/`+0x2ac10` (their failure
conditions) or a differential state dump against a physical-GPU run.

Note (same day): a control experiment was attempted - Xorg (GLX path) on
the same boot died in `Failed to initialize DMA` / `Failed to allocate push
buffer`, but that boot had a stalled CUDA client holding the device, so the
result is inconclusive; retry on a clean boot.

## 5. Next steps (for the successor)

1. Return-value trace of `glsi +0x33b80`'s callees (`+0x301b0`'s
   `+0x3a6f0` / `+0x2ac10` results): the first non-zero return there is the
   true failing layer.
2. Compare the glsi device-engine state with the same probe on a physical
   Turing GPU (the host's own 570 driver runs the identical userspace; a
   host-side `egl_frame` run with the real GPU succeeds, so a differential
   dump of the engine records / the `+0xb84` flags word will show what the
   emulated device is missing).
3. The engine records glsi consumes come from the kernel driver's device
   data (engine table + class list). The CUDA path builds the analogous
   records from `GET_ENGINES_V2`; the CUDA engine list on this guest is
   known to be complete for GR + CE (see the private analysis area). The
   glsi path may require an entry (e.g. a specific engine kind or a caps
   bit) that the emulated device does not report.

## 5. Reproduction

```
# in the guest (570.144 stock driver):
gcc -O2 -o /tmp/egl_frame /tmp/egl_frame.c -lEGL -lGLESv2   # tools/guest-diag/egl_frame.c
timeout 90 /tmp/egl_frame      # EGL_CURRENT_FAIL err=0x3003

# under gdb (offsets stable, ASLR disabled):
gdb -q -batch -x egl-gdb11.txt ./egl_frame    # catches the vtable call site
```

Breakpoints used (no-ASLR addresses with eglcore base 0x7ffff5200000,
glsi base 0x7ffff7e2a000):

| address | meaning |
|---|---|
| `0x7ffff5bfe151` | eglcore rejection call `*0x30(vtable)` |
| `0x7ffff7e89480` | glsi engine-init entry (+0x5f480) |
| `0x7ffff7e894e1` | after glsi's first inner call (returns 0xee00001) |