From f03a45bfeff9b95d956fdc504ebbd009ab566cac Mon Sep 17 00:00:00 2001 From: Conan Jeffrey Truong Date: Wed, 29 Jul 2026 12:02:20 -0700 Subject: [PATCH] docs(executorch): note the single-CUDA-stream requirement for coalesced .pte A coalesced TensorRT + CUDA .pte runs on multiple backend delegates, each of which enqueues async GPU work on its own CUDA stream by default. Because separate streams are unordered, a consuming delegate can read a producing delegate's output before it is ready -- an intermittent race that shows up as wrong results or an illegal memory access. Document that unifying all delegates onto one caller-provided CUDA stream is the runner's responsibility, and that a runner reading delegate outputs between calls (e.g. an autoregressive decode loop) must synchronize that stream before reading. --- .../runtime_performance/saving_models.rst | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docsrc/user_guide/runtime_performance/saving_models.rst b/docsrc/user_guide/runtime_performance/saving_models.rst index 103cf19025..e517cd4e5b 100644 --- a/docsrc/user_guide/runtime_performance/saving_models.rst +++ b/docsrc/user_guide/runtime_performance/saving_models.rst @@ -277,6 +277,29 @@ must be pointed at those data files to load them. ``.pte`` into the same directory overwrites the blob and the first ``.pte`` will fail to load. Save each coalesced model into its own directory. +**Running a coalesced .pte: use a single CUDA stream** + +A coalesced ``.pte`` runs on more than one backend delegate (the TensorRT delegate +plus the CUDA backend). By default each backend enqueues its GPU work on its *own* +CUDA stream, and delegate execution is asynchronous -- a delegate returns after +*enqueuing* its work, not after it completes. Because separate CUDA streams are not +ordered relative to one another, at a delegate boundary the consuming delegate can +begin before the producing delegate's writes have finished, reading incomplete +data. This is a race: it is intermittent and can surface as wrong results or an +illegal memory access. + +The runtime does not impose a shared stream across delegates, so it is the +**runner's responsibility** to run all delegates on one CUDA stream. Create a +single stream and, for the duration of execution, direct every backend to use it +(each backend exposes a caller-stream hook). All GPU work is then enqueued in order +and every cross-boundary dependency is satisfied, while execution stays +asynchronous. + +If the runner reads a delegate's outputs between calls (for example, an +autoregressive decode loop), synchronize the shared stream before reading: the +work may still be in flight when ``execute()`` returns, and a host-side copy on +the default stream will not wait for a non-blocking stream. + Saving torch.compile models -----------------------------