diff options
| -rw-r--r-- | STATE-CURRENT.md | 64 | ||||
| -rw-r--r-- | docs/04-implementation/04-real-render-walls.md | 115 | ||||
| -rw-r--r-- | kernel/gsp_sema_service.c | 678 | ||||
| -rw-r--r-- | kernel/main.c | 41 |
4 files changed, 810 insertions, 88 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 2eb02d61..d9ba69a 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,37 +1,41 @@ # Open-IOV Current State -> **Snapshot: 2026-10-24 ~20:40 (the host-side operand-resolution gap -> is closed and the shadow anchor is content-gated — r635 tracks REAL -> written content per shadow page (`r->content` on every landed write) -> and uses it as the operand anchor: a client's own H2D bytes that took -> the shadow lane anchor their span and arm a launch-level content -> anchor, so an answerless output candidate (the buffer the client never -> wrote) is served from a bounded synthetic page set (1024/candidate, -> 4096/launch, three-quarter-table guard) instead of failing the launch -> closed; trusted pin/walk/ledger answers keep the r633 budgets and no -> longer arm the cross mint, and a synthetic page can never re-anchor. -> The first draft of the change self-fed on its own minted pages and -> filled the shared shadow table on a green boot (6144/8192 measured, -> later clients crawled behind it); the content gate restored zero -> minted pages on a green boot and the historical green shape. Fresh -> r636 boot battery: 9 OK / 2 clean FAIL / 1 HANG — the best pass count -> repeated with strictly better failure modes: streams_events converted -> its historical hang into a clean client-visible `CUDA_ERROR` via the -> auto-fault, and streams2's red chain is now measured end-to-end (its -> H2D destinations are replay-denied, the GR forward defers on the CE -> stream-order watermark, the launch is refused as degenerate and the -> refusal faults cleanly; `STREAMS n=32768 bad=65534`). Remaining: the -> streams2 CE-order chain (a copy-pipeline class, not the fill), the -> first-wait crawl class (smiley/memset2d spin in libcuda before -> submission), used-boot allocation exhaustion (roundtrip/inplace_scale -> `CUDA_ERROR out of memory` after thirteen clients), then 15/15 + soak. -> GL runs on the guest GPU (next wall = the FBO class); the Looking -> Glass frame path is proven end-to-end (guest CUDA render -> KVMFR +> **Snapshot: 2026-11-06 ~01:00 (the guest's REAL Vulkan pipeline now +> renders on the emulated vGPU: the first `VKFRAME seq=1..3` / +> `VKFRAME_DONE frames=3 640x480 saved=3` / `VKFRAME_IDLE ok` runs +> end-to-end — instance, device, allocation, image/view/buffer/shader/ +> descriptor/pipeline/command-buffer/fence, queue submit, fence release +> and frame readback all complete with a zero-touch service pipeline. +> The chain of fixes behind it: the push-stream parser now accepts the +> driver's real shapes (the `0x0001fff0` SET_SUBDEVICE_MASK bracket and +> zero NOP gaps — the class that silently skipped every large push), the +> fence fold (`sema_gr_ahead`) sweeps the pre-built GPFIFO continuation +> beyond GP_PUT and applies the submit's INC-5 RELEASE groups +> forward-only, the host-triggered VAS page-directory anchoring verifies +> the parked `COPY_SERVER_PDES` candidate against a content-located ring +> pair so the walk resolves every push VA before its first read, the +> COMPR+2MB VidHeap class is downgraded at the ioctl so +> `vkAllocateMemory` succeeds, the SEC2 scrubber's finish payload is +> authored by the service, the `0x9096` VidHeap enumeration and the +> `FIFO_GET_ENGINE_CONTEXT_PROPERTIES` ZCULL geometry are answered, and +> the C5C0-section -> vendor-launch translator found the real dispatch +> (a `SET_INLINE_QMD_ADDRESS_B` + 64-word run building QMD `0x4b85f00`, +> grid 40x30, plus the C5B5 copy-out). The frames are still BLACK: the +> driver delivers SASS/descriptors through its MME program + inline-to- +> memory stream (decoded in full; the payload rides the next GPFIFO +> entry as a source-view call), and the utility/copy-channel +> `LOAD_INLINE_DATA` arm that must materialize those writes is the +> current frontier (arbitration + `sema-utildec` witness landed). CUDA +> meanwhile: the best fresh-boot battery is 8 pass / 3 fail / 1 hang +> (the Vulkan work does not regress CUDA; always battery-first on a +> fresh boot); GL runs on the guest GPU (next wall = the FBO class); the +> Looking Glass frame path is proven end-to-end (guest render -> KVMFR > ivshmem -> host PNG frame); two guests with 512M + 1GiB emulated VRAM > coexist at boot level; the 0.9.9 release-preview demo video v1 is -> recorded and synced to files.vgpu.io. Frontier = 15/15 + multi-round -> soak, events A/B, GZDOOM (GL/Vulkan), two-guest concurrent CUDA, -> longevity/video v2, Windows guests).** +> synced to files.vgpu.io (the updated video with GZDOOM working is the +> next publish). Frontier = non-black frames, then Xorg -> GZDOOM +> (GL/Vulkan) -> KVMFR/LG frames -> CUDA 15/15 + soak -> the perfect +> 1.0.0 video -> Windows guests).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md index 4298772..56b5e11 100644 --- a/docs/04-implementation/04-real-render-walls.md +++ b/docs/04-implementation/04-real-render-walls.md @@ -219,3 +219,118 @@ shape the CUDA path already executes) from the section's fields and data runs, and forwards it with the existing operand/relocation machinery. The method vocabulary overlaps the physical prelude's existing knowledge (`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known. + +## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate + +**The arm was dead by branch order.** The utility channel's engine-side +uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and +their operand group (`0x188/0x18c` destination, `0x180` line length, +`0x184` line count) was parsed only by an `else if (subch == 1 || subch == +2)` branch that could never run: the preceding `if (xlate && subch >= 2)` +TURING-copy parser also matches subchannel 2 and swallowed the group +unread. With `have_out`/`line_len` unset, the `0x1b4 +LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now +parsed first for subchannel 1|2; the arm reaches its write decision and +authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem +0x1c0000). + +**The payloads are stably zero - and that is the driver's zero buffer.** +Four independent reads (the sweep's channel path, the direct +registered-VAS walk - 22k attempts, never non-zero - the guest's own +`/proc/<pid>/mem` view of the client's mapping, and the guest's +`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a +gdb hardware write-watchpoint that never fires agree: the driver never +writes the payload chunks for these entries. The 256 B size equals +`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()` +uploads exactly 256 B of zeros through this path. The arm therefore +applies zero payloads under a variant once-key (`h | 1`) - idempotent over +an empty destination - while a later non-zero payload for the same entry +still lands under its own key. + +**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900` +resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is +not delivered by any decoded entry. The utility channel's ring holds +exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA - +both the zero objects above. The Vulkan (NVN) download path uploads via a +separate download state and points its destination view at the pitch-pool +allocation, so the SASS/constants i2m shape is a different destination +family than the observed zero-buffer uploads. Next steps: find the SASS +upload shape (pitch-pool destination, possibly on another channel or a +non-LID copy) and then the `n_data` descriptor/operand wiring +(section 8); with both, the physical forward can execute the dispatch. + +## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem + +**The ring is in vidmem, and its content varies per client.** The +utility channel's ring VA (0x4020000) resolves through the service's +learned map to a **vidmem** offset (PA 0xa0000, next to the channel's +USERD at 0x8fe00), and the pump's own entry read shows the live ring's +content: on the r750-era clients `entry[0] va=0x120100000 len=0x28` +(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000 +va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000) +confirms exactly one entry there. + +**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are +from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA) +can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an +EARLIER client's ring - a coherent 14-entry ring whose LID uploads target +the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000, +dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and +12288 B). Those entries are real, but they belong to a dead session: the +payload chunks they point at (VA 0x4250000/0x4250110 -> the session's +1 MB staging BO) read zero through EVERY path - the sweep's channel read, +the direct registered-VAS walk, the learned map, the guest's own +`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run +high-frequency poller of every 1 MB BO the client maps, and a gdb +hardware write-watchpoint that never fires - because the session ended +and the BO's pages were freed/reused. **The payload is not missing from a +live upload; it is a stale-session artifact.** + +**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on +the same boot show the arm working with real payloads: +`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378 +tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0 +dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the +push and holds a real method stream, the arm materializes it into the +high vidmem family, and the CUDA battery passes. + +**Next steps (ranked).** +1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a + client run and decode its entries - that is the live upload path; the + sweep's `svc_read_va` source preference must not answer with a stale + RAM page (verify the ring page by content: the entries must lie in the + client's current VA allocations, and the ring must be the same page the + pump reads). +2. With the live ring decoded, the SASS upload shape follows; the + descriptor/operand wiring (`n_data=0`) remains the second gate. +3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets + were exhausted early in multi-client boots, hiding later rows; make + them per-session for diagnosis. + +## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not + +`dump_vidmem map` (new: one line per 1 MB block with non-zero content, +`nz` = non-zero dwords) on a fresh boot, same module: + +* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up: + `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`, + `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`, + `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's + uploads (LID i2m writes with real payloads + CE copies) materialize in + the emulated vidmem. +* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14 + baseline dwords (the fence/counter words at 0xfee88, the GR param bank + at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads + reaches the emulated vidmem at all.** + +That is the frontier in one measurement: the Vulkan (NVN/GL) driver's +upload path is not among the writes the emulation can see. Combined with +section 10 (the utility ring lives in vidmem via the learned map but the +sweep can read a stale RAM page for it; the live ring holds a single +`0x120100000/40` entry, not the 14-entry LID ring), the next session must: +(1) capture the LIVE ring through the same source the pump uses while the +client is up and decode its entries (the 14-entry LID ring is a +stale-session page); (2) trace where the live client's upload writes go +(the map shows they are NOT in the first 256 MB of the shadow - check the +guest's own view of the client's staging BOs during the upload window); +(3) the descriptor/operand wiring (`n_data=0`) after that. diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index 8fd602c..6f005b5 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -1866,15 +1866,48 @@ module_param_named(sema_vas_cand_ms, svc_vas_cand_age_ms, uint, 0644); * SESSION 127 C5C0 translator gate. When 1, the sweep additionally * translates continuation entries into launch work (tail-aware method * parse, inline-QMD capture, LOAD_INLINE_DATA materialization, C5B5 - * copy-out capture, candidate-owned release withholding). Default 0 keeps - * the historical sweep behaviour for the CUDA batteries; the Vulkan - * translator cycles pass sema_gr_xlate=1. Flip the default once the CUDA - * battery is proven clean with it on. + * copy-out capture, candidate-owned release withholding). + * + * SESSION 130 re-test of the r727 "translator regressions" (translator on, + * utility sweep OFF, r746): victory_saxpy and victory_math now PASS with + * checksums identical to the gate-off baseline (saxpy + * csum=981440890.500000), so the saxpy/math half of the r727 regression + * is FIXED (the SESSION 129/130 sweep changes); victory_mail PASSES. The + * remaining xlate-on regression is victory_multikernel: its launch is + * captured and submitted to the physical channel but the forward times + * out repeatedly (`GR LAUNCH FAILED: timeout after 2000 ms + * (qmd_rel=0xc0ffee01 want=0x600df00d ...) h=0x5c000019 + * push=0x20043c394`, `hold active`), so the client hangs - the default + * stays 0 until that forward path is fixed. The UTILITY SWEEP + * (`sema_gr_util_scan`) is the OTHER regression source: at 24 it broke + * saxpy (`bad=1047527`) and hung math on r745, so it stays opt-in + * (default 0) until its channel reads prefer the direct walk over pins. */ static bool gsp_sema_gr_xlate; module_param_named(sema_gr_xlate, gsp_sema_gr_xlate, bool, 0644); MODULE_PARM_DESC(sema_gr_xlate, "C5C0 translator: sweep continuation entries into launch work (default 0 = releases-only sweep)"); +/* + * SESSION 128 utility-channel inline-download materialization + * (sema_gr_util_scan, default 0 = off): the driver's non-GR upload + * channels (the MEM2MEM/A140 + 2D utilities, registered but with an + * unresolved USERD so the pump's put-bounded entry loop never decodes + * them) carry the engine-side code/data downloads: the + * GRCE/INLINE_TO_MEMORY operand group on the mem2mem subchannel (0x180.. + * 0x18c), LAUNCH_DMA (0x1b0) and a LOAD_INLINE_DATA (0x1b4) run whose + * payload the engine fetches from the next GPFIFO entry (the + * source-view call). Without this pass those destinations stay zero in + * the vidmem shadow - observed: the Vulkan client's code-heap/SASS + * upload (12 KB) and command-stream staging (256 B) never land, so the + * forwarded launch has no program content. Bounded: scan up to N + * entries per channel per pass, materialize once-only keyed by + * (push_va, content hash, dst, bytes) via the same + * svc_grce_inline_write the GR sweep uses. Requires sema_gr_xlate. + */ +static u32 gsp_sema_gr_util_scan; +module_param_named(sema_gr_util_scan, gsp_sema_gr_util_scan, uint, 0644); +MODULE_PARM_DESC(sema_gr_util_scan, + "Utility-channel i2m sweep: entries scanned per pass (default 0 = off; requires sema_gr_xlate)"); MODULE_PARM_DESC(sema_vas_cand_ms, "Parked-VAS-candidate age-out (ms, default 15000; 0 = never)"); static void svc_vas_candidates_age_out(struct nv_vgpu_device *vgpu) @@ -24057,6 +24090,38 @@ static u64 svc_gr_ahead_skips; static u64 svc_gr_ahead_owned; /* SESSION 127: bounded witness for the sweep-caller's launch request. */ static u32 svc_gr_ahead_req_log; +/* SESSION 128: inline (LOAD_INLINE_DATA) writes authored by the sweep. */ +static u64 svc_gr_ahead_inline_writes; + +/* + * SESSION 129 util-channel witness: the utility channel's subchannel-2 + * LOAD_INLINE_DATA arm never fired (writes=0 on r737) while the same arm + * fired for in-push payloads on the GR channel - i.e. the failure sits in + * the split (tail-served) shape. This struct carries the per-entry story + * out of the translator for one bounded, deduplicated witness line: + * which read source answered (attempt A = the channel path, 1 = the direct + * registered-VAS walk), whether the strict parse accepted the entry, and + * what the LID arm did (saw/fired/refused a still-zero payload). + */ +struct svc_util_wit { + u32 p_src; /* attempt that won: 0 = channel, 1 = VAS walk, 0xff none */ + u32 pa_ok; /* attempt-A parse (method count) */ + u32 pb_ok; /* attempt-B parse (method count) */ + u32 t_src; /* tail read source: 0 = channel, 1 = walk, 0xff none */ + u32 t_len; /* tail length served */ + u32 t0, t1; /* SESSION 130: first two words of the served tail */ + bool saw_lid; /* a LOAD_INLINE_DATA arm was reached */ + bool fired; /* ... and authored its write */ + bool zero; /* ... but the tail payload was still all-zero (pending) */ + u32 dst; + u32 line_len; + u32 line_lc; + u32 count; +}; +static struct svc_util_wit svc_util_wit; +static u64 svc_util_pending_zero; +static u32 svc_util_dec_log; +static struct { u64 va; u64 h; } svc_util_dec_keys[32]; /* * SESSION 127 C5C0 translator, bounded tail materialization buffer: a @@ -24065,10 +24130,98 @@ static u32 svc_gr_ahead_req_log; * entry plus a dedicated data slot entry). 4 KB covers the observed * param-bank write (1568 B) in full and the first page of the big uploads. */ -#define SVC_AHEAD_TAIL_INLINE_MAX 4096 +#define SVC_AHEAD_TAIL_INLINE_MAX 32768 static u8 svc_ahead_inline_buf[SVC_AHEAD_TAIL_INLINE_MAX]; static u64 svc_ahead_inline_trunc; +/* + * SESSION 128: one-copy materialization bound for the utility/CE sweep. + * The engine-side copy-in the sweep replays (a TURING_DMA_COPY_A pitch + * copy from a client-owned staging buffer into the vidmem shadow) moves + * the driver's code/constant uploads; the observed SASS/code-heap class is + * tens of KB, so 4 MB is a generous cap that still refuses a bogus + * geometry before it can walk the shadow. + */ +#define SVC_AHEAD_COPY_IN_MAX (4ULL << 20) + +/* + * SESSION 128: materialize one TURING_DMA_COPY_A pitch copy-in that an + * unprocessed copy-class channel carried - src bytes live in the client's + * staging buffer (guest RAM, resolved through the trusted walk like any + * operand page), dst is the vidmem page the driver staged the upload for + * (code heap / constant store). This is the emulated engine write for the + * copy the real CE would perform; it is once-only keyed by (push_va, + * content hash, dst, bytes) so re-scans of a stable ring are free. + */ +static u64 svc_ahead_copy_in_skip; +static void svc_ahead_copy_in(struct nv_vgpu_device *vgpu, u32 ch_h, + u64 push_va, u64 hash, u64 src_va, u64 dst_va, + u32 row, u32 rows, u32 dma) +{ + static u8 cb[65536]; + u64 bytes = (u64)row * (rows ? rows : 1); + u64 done = 0; + u64 spa = 0, dpa = 0; + int sap = SVC_APERTURE_SYS, dap = SVC_APERTURE_SYS; + int rc = 0; + bool src_ok, dst_ok; + + if (!bytes || src_va < SVC_LOW_REL_VA_LO || src_va >= SEMA_VA_HI || + dst_va < SVC_LOW_REL_VA_LO || dst_va >= SEMA_VA_HI) + return; + /* + * Only the upload direction is materialized: the source must be a + * resolved client page in sysmem/guest RAM (the staging buffer the + * client CPU-wrote), and the destination must be vidmem (resolved to + * the FB aperture) or a not-yet-resolved VA whose bytes the VA + * shadow will hold. A resolved SYS destination is a copy-OUT (the + * engine reading vidmem into a client buffer): replaying it here + * would overwrite the client's buffer with shadow garbage before the + * producing launch ran, so it is skipped - the GR channels' copy-out + * machinery owns that direction. + */ + src_ok = !svc_lookup(vgpu, src_va, &spa, &sap) && sap != SVC_APERTURE_FB; + dst_ok = svc_lookup(vgpu, dst_va, &dpa, &dap) || + dap == SVC_APERTURE_FB; + if (!src_ok || !dst_ok) { + svc_ahead_copy_in_skip++; + if (svc_ahead_copy_in_skip <= 8) + pr_info("sema-utilsweep: copy-in skipped (src_ok=%d dst_ok=%d) push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx\n", + src_ok, dst_ok, push_va, src_va, dst_va, + bytes); + return; + } + if (bytes > SVC_AHEAD_COPY_IN_MAX) { + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-utilsweep: copy-in too large push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx cap=0x%llx\n", + push_va, src_va, dst_va, bytes, + SVC_AHEAD_COPY_IN_MAX); + return; + } + if (svc_inline_once_seen(push_va, hash, dst_va, (u32)bytes)) + return; + svc_inline_once_store(push_va, hash, dst_va, (u32)bytes); + while (done < bytes) { + u32 chunk = bytes - done > sizeof(cb) ? + (u32)sizeof(cb) : (u32)(bytes - done); + + rc = svc_read_va(vgpu, src_va + done, cb, chunk); + if (rc) { + rc = -EIO; + break; + } + rc = svc_grce_inline_write(vgpu, dst_va + done, cb, chunk, + ch_h); + if (rc) + break; + done += chunk; + } + svc_gr_ahead_inline_writes++; + pr_info("sema-utilsweep: copy-in src=0x%llx -> dst=0x%llx bytes=0x%llx dma=0x%x done=0x%llx rc=%d h=0x%x\n", + src_va, dst_va, bytes, dma, done, rc, ch_h); +} + /* One 4-byte word of a method stream that may continue into the tail. */ static u32 svc_ahead_w32(const u8 *buf, u32 len, const u8 *tail, u32 taillen, u32 off) @@ -24172,7 +24325,7 @@ done: */ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch, const u8 *buf, u32 len, const u8 *tail, - u32 taillen, u64 push_va) + u32 taillen, u64 push_va, bool inline_only) { static struct svc_push_state stq; /* service thread only */ static u32 dw[66]; /* QMD window words */ @@ -24236,7 +24389,7 @@ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch * C5C0 inline-QMD window (mirrors the main decode's intercept): * only accumulate once the run declares itself a QMD window. */ - if (xlate && forward && subch >= 1 && + if (xlate && forward && !inline_only && subch >= 1 && addr >= SVC_QMD_ADDR_A && addr <= SVC_QMD_DATA + (SVC_QMD_WORDS - 1) * 4 && (stq.qmd_have || @@ -24269,6 +24422,8 @@ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch u32 cap = count * 4; u32 bytes, k, words; u64 h = svc_push_hash(buf, len); + bool tail_served = end > len; + bool refuse_zero = false; if (nbytes > cap) nbytes = cap; @@ -24288,10 +24443,69 @@ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch taillen, off + 4 + 4 * k) }, 4); - if (bytes && !svc_inline_once_seen(push_va, h, off_out, - bytes)) { + /* + * SESSION 129: a tail-served payload whose head is still + * all zeros means the tail source answered with an empty + * page (a stale pin shadow, or the driver has not staged + * the bytes yet) - authoring zeros over the destination + * would corrupt the code heap. Leave the write pending + * (no once-store, the sweep retries every pass) until a + * nonzero payload arrives. In-push payloads are never + * gated: the bytes are part of the flushed submit. + */ + if (inline_only && tail_served && bytes && + !memchr_inv(svc_ahead_inline_buf, 0, + min_t(u32, bytes, 16))) + refuse_zero = true; + if (inline_only) { + svc_util_wit.saw_lid = true; + svc_util_wit.dst = (u32)off_out; + svc_util_wit.line_len = line_len; + svc_util_wit.line_lc = line_count; + svc_util_wit.count = count; + if (refuse_zero) + svc_util_wit.zero = true; + } + if (refuse_zero) { + /* + * SESSION 130: a zero-headed tail-served + * payload is applied under a VARIANT once-key + * (h | 1) instead of being refused forever: + * the destination is currently empty, so + * authoring zeros is idempotent and correct + * for the driver's internal zero buffer (the + * observed 256 B utility i2m = exactly + * GF100_ZERO_BUFFER_SIZE), while a later + * non-zero payload for the same entry still + * matches its own (h) key and lands - the + * refusal can no longer stall the arm + * (r737/r739: `writes=0` with the payload + * page stably zero). + */ + svc_util_pending_zero++; + if (bytes && !svc_inline_once_seen(push_va, + h | 1, + off_out, + bytes)) { + svc_inline_once_store(push_va, h | 1, + off_out, bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, + off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + } else if (bytes && !svc_inline_once_seen(push_va, h, + off_out, + bytes)) { svc_inline_once_store(push_va, h, off_out, bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; (void)svc_grce_inline_write(vgpu, off_out, svc_ahead_inline_buf, bytes, @@ -24310,12 +24524,19 @@ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch &svc_grce_dst_ring[svc_grce_dst_ring_next % SVC_GRCE_DST_MAX]; - rec->dst_va = off_out; - rec->len = min_t(u32, bytes, - GSP_BRIDGE_INLINE_BYTES); - memcpy(rec->data, svc_ahead_inline_buf, - rec->len); - svc_grce_dst_ring_next++; + /* + * A refused (still-zero) tail payload is not + * recorded either: the next pass re-reads the + * tail and the real bytes land then. + */ + if (!refuse_zero) { + rec->dst_va = off_out; + rec->len = min_t(u32, bytes, + GSP_BRIDGE_INLINE_BYTES); + memcpy(rec->data, svc_ahead_inline_buf, + rec->len); + svc_grce_dst_ring_next++; + } } if (end > len) break; @@ -24323,8 +24544,49 @@ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch continue; } - /* TURING_DMA_COPY_A operand runs (subch >= 2). */ - if (xlate && subch >= 2) { + /* + * GRCE / INLINE_TO_MEMORY operand group (0x180..0x18c), for + * subchannel 1 AND 2. The mem2mem subchannel (2, the + * A140/KEPLER_INLINE_TO_MEMORY_B upload lane the utility + * channels drive) carries the same operand methods as the + * compute subchannel (1). This group MUST be parsed BEFORE + * the TURING-copy branch below: that branch's `subch >= 2` + * guard also matches subchannel 2 and would swallow the + * 0x180..0x18c methods unread, leaving have_out/line_len + * unset - the exact reason the utility channel's subch-2 + * LOAD_INLINE_DATA arm never fired (r737/r739 witness: + * `lidn=1 subch=2 ... saw=0 dst=0 ll=0`). + */ + if (xlate && (subch == 1 || subch == 2) && + addr >= 0x180 && addr <= 0x18c && count) { + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x180: + line_len = vv[k]; + break; + case 0x184: + line_count = vv[k]; + break; + case 0x188: + off_out = (off_out & 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x18c: + off_out = (off_out & + ~0xffffffffULL) | vv[k]; + have_out = 1; + break; + default: + break; + } + } + } else if (xlate && subch >= 2) { + /* TURING_DMA_COPY_A operand runs (subch >= 2). */ u32 k, n = count > 4 ? 4 : count; u32 vv[4] = { w0, w1, w2, w3 }; @@ -24394,52 +24656,35 @@ static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch * movement, nothing to record). */ last_dma = dma; - if (forward && (dma & 0x3) && have_in && - have_out && line_len && line_count && + if ((dma & 0x3) && have_in && have_out && + line_len && line_count && off_in && off_out) { - svc_gr_copy_pend_add(ch->h_object, - push_va, svc_push_hash(buf, - len), - off_in, off_out, - line_len * 4, line_count, - pitch_out ? pitch_out : + if (inline_only) + svc_ahead_copy_in(vgpu, + ch->h_object, push_va, + svc_push_hash(buf, + len), + off_in, off_out, line_len * 4, - pitch_in, dma, last_remap); - cp_subch = subch; - } - } - } else if (xlate && subch == 1 && addr >= 0x180 && - addr <= 0x18c && count) { - u32 k, n = count > 4 ? 4 : count; - u32 vv[4] = { w0, w1, w2, w3 }; - - /* GRCE / INLINE_TO_MEMORY operand group (0x180..0x18c). */ - for (k = 0; k < n; k++) { - switch (addr + 4 * k) { - case 0x180: - line_len = vv[k]; - break; - case 0x184: - line_count = vv[k]; - break; - case 0x188: - off_out = (off_out & 0xffffffffULL) | - ((u64)(vv[k] & 0x1ffff) - << 32); - have_out = 1; - break; - case 0x18c: - off_out = (off_out & - ~0xffffffffULL) | vv[k]; - have_out = 1; - break; - default: - break; + line_count, dma); + else if (forward) { + svc_gr_copy_pend_add(ch->h_object, + push_va, svc_push_hash(buf, + len), + off_in, off_out, + line_len * 4, + line_count, + pitch_out ? pitch_out : + line_len * 4, + pitch_in, dma, + last_remap); + cp_subch = subch; + } } } } - if (addr == HOST_M_SEM_ADDR_LO && count >= 5) { + if (!inline_only && addr == HOST_M_SEM_ADDR_LO && count >= 5) { u64 va; va = ((u64)(w1 & 0xff) << 32) | @@ -24548,7 +24793,7 @@ static void svc_gr_ahead_sweep(struct nv_vgpu_device *vgpu, } swept++; nrel += svc_gr_ahead_push(vgpu, ch, abuf, rlen, nbuf, - tail_len, va); + tail_len, va, false); } svc_gr_ahead_entries += swept; svc_gr_ahead_rels += nrel; @@ -24561,6 +24806,305 @@ static void svc_gr_ahead_sweep(struct nv_vgpu_device *vgpu, } } +/* + * SESSION 129: read a VA straight through the registered-VAS walks only + * (newest registration first), one page at a time - the exact path + * walk_probe and sema_probe prove. The util sweep's channel read can be + * shadowed by a stale injected pin / window guess for the utility VAs + * (observed r737: the subchannel-2 LID arm never fired although the push + * page content is real and the VAS walk resolves both the methods entry + * and its payload slot); this helper is the arbitration fallback that + * bypasses those sources entirely. + */ +static int svc_util_walk_read(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len) +{ + u8 *p = buf; + + while (len) { + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + size_t off = va & (SEMA_PAGE_SIZE - 1); + size_t chunk = min_t(size_t, len, SEMA_PAGE_SIZE - off); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (svc_walk_vaspaces(vgpu, page, &pa, &ap)) + return -ENOENT; + if (svc_read_ap(vgpu, pa + off, svc_ap_of(ap), p, chunk)) + return -EIO; + p += chunk; + va += chunk; + len -= chunk; + } + return 0; +} + +/* + * SESSION 129: does this entry window carry a LOAD_INLINE_DATA header? Used + * by the bounded util-sweep witness to select only the entries that can + * exercise the arm (one per distinct content, so a stable ring logs once). + */ +static u32 svc_util_lid_scan(const u8 *buf, u32 len, u32 *first_subch) +{ + u32 k, n = 0; + + *first_subch = 0xff; + for (k = 0; k + 4 <= len; k += 4) { + u32 hdr; + + memcpy(&hdr, buf + k, 4); + if (((hdr >> 29) & 0x7) == 3 && + ((hdr & 0x1fff) << 2) == GRCE_M_LOAD_INLINE_DATA) { + if (*first_subch == 0xff) + *first_subch = (hdr >> 13) & 0x7; + n++; + } + } + return n; +} + +/* + * SESSION 128: utility-channel inline-download materialization. + * + * The driver's upload/utility channels (the MEM2MEM/A140 + C5B5 + 2D + * composite: observed h=0xbeef0100 with gpfifo=0x4020000) are registered + * with an unresolved USERD (userd_ok=0), so the put-bounded entry loop in + * svc_pump_channels never decodes their pushes and the LOAD_INLINE_DATA + * downloads they carry (the client's code-heap/SASS upload, command-stream + * staging) never land anywhere the emulation can see: the destinations + * stay zero in the vidmem shadow. + * + * This pass scans a bounded window of the channel's GPFIFO directly (the + * gpfifo VA resolves through the trusted walk even when the USERD does + * not) and feeds each entry + its successor (the source-view call carrying + * the payload) through the proven sweep decoder in inline-only mode: the + * GRCE/INLINE_TO_MEMORY operand group (0x180..0x18c on subch 1/2), + * LAUNCH_DMA and the LOAD_INLINE_DATA run are materialized by + * svc_grce_inline_write; releases, QMD windows and copy-pend records are + * skipped so nothing about the client's fence/completion state changes. + * Once-only keyed by (push_va, content hash, dst, bytes), so a stable ring + * costs only the reads. Requires sema_gr_xlate; bounded by + * sema_gr_util_scan entries per channel per pass (0 = off). + * + * SESSION 129 read arbitration: attempt A is the channel path (unchanged); + * when A fails to read, or reads content the strict parse rejects, attempt + * B re-reads the push and the tail through svc_util_walk_read and the + * parse arbitrates - a source that yields a method stream wins. The r737 + * evidence (arm never fired while the content is real) pointed at exactly + * this: a shadowed channel read consumed the entry without a parseable + * buffer while the walk resolves the true pages. + */ +static void svc_gr_util_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ +/* + * The observed utility-channel pushes are the ~17 KB setup block and the + * two split uploads (256 B command stream + 12 KB code/heap payload); a + * 64 KB window covers every one of them with room to spare while keeping + * the two static buffers modest. + */ +#define SVC_UTIL_BUF (64 * 1024) + static u8 ubuf[SVC_UTIL_BUF]; + static u8 nbuf[SVC_UTIL_BUF]; + static u8 bbuf[SVC_UTIL_BUF]; + static u8 tbuf[SVC_UTIL_BUF]; + static u64 svc_gr_util_entries; + static u64 svc_gr_util_writes; + static u32 svc_gr_util_log; + static u64 svc_util_tail_alt; /* SESSION 130: tails re-read through + * the direct walk because the + * channel read was zero-headed */ + static u64 svc_util_tail_try; /* ... and the walk attempts (alt is + * the subset that yielded non-zero) */ + u32 i, maxent = min_t(u32, gsp_sema_gr_util_scan, 64); + u32 swept = 0, nmade = 0; + + if (!gsp_sema_gr_xlate || !gsp_sema_gr_util_scan || + !ch->gpfifo_va || !ch->gpfifo_entries) + return; + for (i = 0; i < maxent && i < ch->gpfifo_entries; i++) { + u8 ent[SVC_GPFIFO_ENTRY_SIZE], nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen; + u64 va, before; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + continue; /* unwritten slot */ + if (!va || !len || len > SVC_PUSH_MAX) + continue; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + continue; + rlen = min_t(u32, len, sizeof(ubuf)); + /* + * SESSION 129 read arbitration + bounded witness (see the + * sweep's header comment). Attempt A: the channel path, + * unchanged. Attempt B: the direct registered-VAS walk, used + * when A cannot read or cannot parse; the strict parse then + * arbitrates which content is the method stream. + */ + { + u32 push_rc, parse_a = 0, parse_b = 0; + u32 tl_a = 0, tl_b = 0; + u32 lidn, lidsc = 0xff, slot; + u64 nva = 0; + u32 nlen = 0; + u64 hkey; + u8 *pbuf = ubuf; + bool ut = false, seen = false; + + push_rc = svc_read_va_push(vgpu, ch, va, ubuf, + rlen); + /* + * The payload of a trailing LOAD_INLINE_DATA run rides + * the next GPFIFO entry (the driver's source-view + * GPFIFO call, observed beef0100 idx00->idx01 and + * idx08->idx09): read it as the tail exactly like + * svc_gr_ahead_sweep does. + */ + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)((i + 1) % ch->gpfifo_entries) * + SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent)) == 0) { + u32 ne0, ne1; + + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + } + if (!push_rc && nva && nlen && nlen < SVC_PUSH_MAX) { + tl_a = min_t(u32, nlen, sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, nbuf, + tl_a)) + tl_a = 0; + } + if (!push_rc) + parse_a = svc_ahead_parse(ubuf, rlen, nbuf, + tl_a, &ut); + if (push_rc || !parse_a) { + bool bpush = false; + + if (nva && nlen && nlen < SVC_PUSH_MAX) { + u32 want = min_t(u32, nlen, + sizeof(nbuf)); + + if (!svc_util_walk_read(vgpu, nva, + nbuf, + want)) + tl_b = want; + } + if (!svc_util_walk_read(vgpu, va, bbuf, + rlen)) { + bpush = true; + parse_b = svc_ahead_parse(bbuf, rlen, + nbuf, tl_b, + &ut); + /* + * A's buffer is authoritative only + * when A actually read it; otherwise + * (or when only B parses) B's walk + * content is the stream. + */ + if (push_rc || + (parse_b && !parse_a)) { + pbuf = bbuf; + tl_a = tl_b; + } + } + if (push_rc && !bpush) + continue; /* unreadable both ways */ + } + /* + * SESSION 130 tail-content arbitration: a tail-served + * payload whose channel head is still all-zero is not + * the staged bytes (the page the channel translation + * points at is empty, or the read is shadowed by a + * stale pin). Try the direct registered-VAS walk - + * the same path attempt B proves - before handing + * the tail to the LID arm, whose zero-headed refusal + * would otherwise stall the upload every pass. + */ + if (tl_a && nva && + !memchr_inv(nbuf, 0, min_t(u32, tl_a, 16))) { + u32 want = tl_a; + + svc_util_tail_try++; + if (!svc_util_walk_read(vgpu, nva, tbuf, + want) && + memchr_inv(tbuf, 0, + min_t(u32, want, 16))) { + memcpy(nbuf, tbuf, want); + svc_util_tail_alt++; + } + } + memset(&svc_util_wit, 0, sizeof(svc_util_wit)); + if (tl_a >= 4) + memcpy(&svc_util_wit.t0, nbuf, 4); + if (tl_a >= 8) + memcpy(&svc_util_wit.t1, nbuf + 4, 4); + before = svc_gr_ahead_inline_writes; + (void)svc_gr_ahead_push(vgpu, ch, pbuf, rlen, nbuf, + tl_a, va, true); + if (svc_gr_ahead_inline_writes != before) + nmade++; + swept++; + /* + * Bounded witness, one line per distinct LID-carrying + * (or unparseable) entry content: which attempt was + * used, the parses, the tail resolution and what the + * LID arm did. + */ + lidn = svc_util_lid_scan(pbuf, rlen, &lidsc); + if (!lidn && parse_a) + continue; + hkey = svc_push_hash(pbuf, rlen); + for (slot = 0; slot < 32; slot++) + if (svc_util_dec_keys[slot].h == hkey && + svc_util_dec_keys[slot].va == va) { + seen = true; + break; + } + if (seen || svc_util_dec_log >= 96) + continue; + slot = svc_util_dec_log & 31; + svc_util_dec_keys[slot].va = va; + svc_util_dec_keys[slot].h = hkey; + svc_util_dec_log++; + pr_info("sema-utildec: h=0x%x i=%u va=0x%llx len=0x%x p_src=%u pa=%u pb=%u tail_va=0x%llx tail=0x%x t8=%08x %08x alt=%llu/%llu lidn=%u subch=%u saw=%d fired=%d zero=%d dst=0x%x ll=%u lc=%u f8=%08x %08x\n", + ch->h_object, i, va, rlen, + pbuf == bbuf ? 1u : 0u, parse_a, parse_b, + nva, tl_a, svc_util_wit.t0, svc_util_wit.t1, + svc_util_tail_alt, svc_util_tail_try, + lidn, lidsc, + svc_util_wit.saw_lid ? 1 : 0, + svc_util_wit.fired ? 1 : 0, + svc_util_wit.zero ? 1 : 0, + svc_util_wit.dst, svc_util_wit.line_len, + svc_util_wit.line_lc, + rlen >= 4 ? *(const u32 *)pbuf : 0, + rlen >= 8 ? *(const u32 *)(pbuf + 4) : 0); + } + } + svc_gr_util_entries += swept; + svc_gr_util_writes += nmade; + if (swept && (nmade || svc_gr_util_log < 24)) { + svc_gr_util_log++; + pr_info("sema-utilsweep: h=0x%x swept=%u writes=%u (total %llu/%llu)\n", + ch->h_object, swept, nmade, svc_gr_util_entries, + svc_gr_util_writes); + } +} + static void svc_pump_channels(struct nv_vgpu_device *vgpu) { if (svc_pdb_disc_pending) { @@ -24871,6 +25415,24 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) jiffies + 27 * HZ; } + /* + * SESSION 128: utility/upload channels carry the engine's + * inline code/data downloads (LOAD_INLINE_DATA on the + * mem2mem subchannel) and the copy-class channels (eng=11/12) + * carry the driver's pitch copy-ins (staging -> code heap / + * constant store). With an unresolved USERD the put-bounded + * entry loop below never decodes them (the !userd_ok branch + * continues before it); with a resolved USERD the main + * decode's LOAD_INLINE_DATA/operand arms still only accept + * subchannel 1, so the subchannel-2 downloads are missed + * either way. Scan every channel's GPFIFO directly here and + * materialize the downloads (see svc_gr_util_sweep); the + * inline-only mode plus the once-only key make the overlap + * with the main loop and the GR sweep a no-op. + */ + if (gsp_sema_gr_util_scan && gsp_sema_gr_xlate) + svc_gr_util_sweep(vgpu, ch); + if (!ch->userd_ok) { if (svc_chan_userd(vgpu, ch, rrc ? 0 : ring_pa, rrc ? -1 : ring_ap, &put, &get, diff --git a/kernel/main.c b/kernel/main.c index 7af236d..7016647 100644 --- a/kernel/main.c +++ b/kernel/main.c @@ -3482,6 +3482,47 @@ static int nv_vgpu_dump_vidmem_set(const char *val, const struct kernel_param *k mutex_unlock(&nv_vgpu_driver.lock); return 0; } + if (!strncmp(buf, "map", 3)) { + /* + * Coarse non-zero map: one line per 1 MB block that holds + * any non-zero dword (SESSION 131: locate the live client's + * uploaded content - the image BO, the code heap, the rings - + * inside the 256 MB aperture without 65536 range dumps). + */ + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = vgpu->bars[1].size; + size_t blk, nblk = (size + (1 << 20) - 1) >> 20; + size_t o; + + if (!base || !size) + continue; + for (blk = 0; blk < nblk; blk++) { + size_t start = blk << 20; + size_t end = start + (1 << 20); + u32 nz = 0, first = 0; + + if (end > size) + end = size; + for (o = start; o + 4 <= end; o += 4) { + u32 v; + + memcpy(&v, base + o, 4); + if (v) { + if (!nz) + first = (u32)(o & 0xFFFFF); + nz++; + } + } + if (nz) + pr_info("vgsp: vidmem map blk 0x%zx nz=%u first=0x%x\n", + start, nz, first); + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return 0; + } comma = strchr(buf, ','); if (comma) |