# StreamDiffusion: A Pipeline-level Solution for Real-time Interactive
Generation

Akio Kodaira1,∗ Chenfeng Xu1,∗ Toshiki Hazama1,∗ Takanori Yoshimoto2 Kohei
Ohno3
Shogo Mitsuhori4 Soichi Sugano5 Hanying Cho6 Zhijian Liu7 Kurt Keutzer1

1UC Berkeley 2University of Tsukuba 3International Christian University
4Toyo University 5Tokyo Institute of Technology 6Tohoku University 7MIT
{akio.kodaira<EMAIL_ADDRESS>
###### Abstract

††∗ denotes equal contribution††This work was done when Toshiki was a remote
intern at UC Berkeley

We introduce StreamDiffusion, a real-time diffusion pipeline designed for
interactive image generation. Existing diffusion models are adept at creating
images from text or image prompts, yet they often fall short in real-time
interaction. This limitation becomes particularly evident in scenarios
involving continuous input, such as Metaverse, video game graphics rendering,
live video streaming, and broadcasting, where high throughput is imperative.
To address this, we present a novel and simple approach that transforms the
original sequential denoising into the batching denoising process. Stream
Batch eliminates the conventional wait-and-interact approach and enables fluid
and high throughput streams. To handle the disparity in frequencies between
data input and model throughput, we design a novel input-output queue for
parallelizing the streaming process. Moreover, the existing diffusion pipeline
uses classifier-free guidance to enforce the generated results conditioned on
the prompts. We point out that current implementations are inefficient due to
the inherent redundancy of negative conditional denoising. To mitigate the
redundant computations, we propose a novel residual classifier-free guidance
(RCFG) algorithm that reduces the number of negative conditional denoising
steps to only one or even zero. Besides, to optimize power consumption, we use
a simple and effective stochastic similarity filtering strategy, which
significantly reduces GPU activation frequency, thereby improving GPU
utilization efficiency. Our batching denoising strategy achieves around 1.5x
speedup compared to the sequential denoising method at different denoising
levels. The proposed (RCFG) leads to up-to 2.05x speedups compared to the
previous conventional classifier-free guidance. The combination of the
proposed strategies and existing mature acceleration tools makes the image-to-
image generation achieve up-to 91.07fps on one RTX 4090 GPU, improving the
throughputs of AutoPipline developed by Diffusers over 59.56x. Beyond the
pipeline efficiency, our proposed StreamDiffusion also significantly reduces
the energy consumption by 2.39x on one RTX 3060 GPU and 1.99x on one RTX 4090
GPU for a static scene input, respectively. The code is available at
https://github.com/cumulo-autumn/StreamDiffusion.

Figure 1: Real-time image-to-image results from camera and screen captures.
The first and second columns show examples of AI-assisted drawing in real-
time, and the third column displays the real-time rendering of 2D
illustrations from 3D avatars. The fourth and fifth columns demonstrate the
real-time camera filter.

##  1 Introduction

Recently, there has been a growing trend in the commercialization of diffusion
models [21, 23, 3] for applications within the Metaverse, as well as in the
realms of online video streaming and broadcasting. These areas require
diffusion pipelines that offer high throughput and low latency to ensure
efficient human interaction. A pertinent example is the use of diffusion
models to create virtual YouTubers. These digital personas should be capable
of reacting in a fluid and responsive manner to user input.

To advance high throughput and real-time interactive capabilities, current
efforts primarily focus on reducing the number of denoising iterations, such
as decreasing from 50 iterations to just a few [16, 17] or even one [28, 13].
The common strategy is to distill the multi-step diffusion models into a few
steps or re-frame the diffusion process with neural Ordinary Differential
Equations (ODE) [14, 26]. Quantization has also been applied to diffusion
models [12, 9] to improve efficiency.

In this paper, we start from an orthogonal direction and introduce
StreamDiffusion, a pipeline-level solution that enables real-time interactive
image generation with high throughput. We highlight that existing model design
efforts can still integrate with our pipeline. Our approach enables the use of
N-step denoising diffusion models while still keeping high throughput and
offers users more flexibility in choosing their preferred models.

We leverage a simple strategy in StreamDiffusion: instead of the original
sequential denoising mode, we batch the denoising steps. We are inspired by
the asynchronous processing in the computer architecture pipeline, and point
out that we do not need to wait for the previous denoising processes to finish
then to start the current denoising step, as shown in Fig. 2. In order to deal
with the synchronous issue of input frequency and U-Net processing frequency,
we leverage a queue strategy to cache the inputs and output. Our pipeline is
different from merely asynchronous processing. We make use of the advantage of
GPU parallelism; thus, we can use one U-Net to denoise a batched noise latent
feature. More importantly, the existing diffusion pipeline incorporated
classifier-free guidance to emphasize the given prompts in the generated
images. Nevertheless, traditional implementations of classifier-free guidance
are plagued by excessive and redundant computational overheads. We introduce
an innovative approach termed as residual classifier-free guidance (RCFG). The
novel technique approximates the negative condition with a virtual residual
noise, which allows us to calculate the negative condition noise only during
the initial step of the process.

Further, we point out that maintaining the diffusion models always in an
interactive mode is energy-consuming as it keeps hitting GPU. To reduce the
energy, we further apply a stochastic similarity filtering strategy. In the
pipeline, we compute the similarities between continuous inputs and determine
whether the diffusion model should process the images based on the similarity.

Experiments demonstrate that our proposed StreamDiffusion can achieve up to
91.07fps for image generation on a one RTX4090 GPU, surpassing the diffusion
Autopipeline from Diffusers team by up to 59.6x. Besides, our stochastic
similarity filtering strategy significantly reduces the GPU power usage by
2.39x on one RTX 3090GPU and by 1.99x on one RTX 4090GPU. Our proposed
StreamDiffusion is a new diffusion pipeline that is not only efficient but
also energy-saving.

##  2 Related works

###  2.1 Diffusion Models

Figure 2: The concept of Stream Batch. In our approach, instead of waiting for
a single image to be fully denoised before processing the next input image, we
accept the next input image after each denoising step. This creates a
denoising batch where the denoising steps are staggered for each image. By
concatenating these staggered denoising steps into a batch, we can efficiently
process continuous inputs using a U-Net for batch processing. The input image
encoded at timestep $t$ is generated and decoded at timestep $t+n$, where $n$
is the number of the denoising steps.

Diffusion models, as introduced in [25, 8], have sparked considerable interest
in the commercial sector due to their high-quality image generation
capabilities and the level of control they offer over the generated images.
These models have been progressively adapted for various applications,
including text-to-image generation [20, 19, 2], image edition [1, 22], and
video generation [4, 5].

A notable advancement in this area is the development of consistent models
[26], which have demonstrated the potential to improve the efficiency of the
sampling process with rarely compromising on image quality. Building on this,
recent studies have focused on enhancing the efficiency of diffusion models by
reducing the number of sampling steps required, further expanding their
practical applicability and efficiency.

###  2.2 Accelerating Diffusion Model

Diffusion models are currently limited by their slow speed in generating
outputs. To address this drawback, several strategies have been developed.
These include methods that don’t require additional training, like using
Ordinary Differential Equation (ODE) solvers, as explored by [26, 14, 15, 17].
Other techniques involve adaptive step size solvers as discussed by [10] and
predictor-corrector methods noted by [25]. The training-based methods are also
proposed, such as optimized discretization techniques [27], neural operator
design [29], and model distillation [24]. Our proposed method distinguishes
itself significantly from the approaches mentioned previously. While earlier
methods primarily focus on the low latency of their individual model designs,
our approach takes a different route. We introduce a comprehensive pipeline-
level solution specifically tailored for high throughputs, which enables
efficient interactive diffusion. Our pipeline is designed to seamlessly
integrate most of the low-latency diffusion models discussed above. By doing
so, we achieve a notable increase in speed within the interactive generation
process.

##  3 StreamDiffusion Pipeline

StreamDiffusion is a new diffusion pipeline aiming for high throughput. It
comprises several key components: the Stream Batch strategy, Residual
Classifier-Free Guidance (RCFG), an input-output queue, Stochastic Similarity
Filter, a pre-computation procedure, and model acceleration tools with a tiny-
autoencoder. We elaborate on the details below.

Figure 3: Virtual residual noise vectors: The orange vectors depict the
virtual residual noise that starts from the PF ODE trajectory and points to
the original input latent $x_{0}$ Figure 4: Input-Output Queue: The process of
converting input images into a tensor data format manageable by the pipeline,
and conversely, converting decoded tensors back into output images requires a
non-negligible amount of additional processing time. To avoid adding these
image processing times to the bottleneck process, the neural network inference
process, we have segregated image pre-processing and post-processing into
separate threads, allowing for parallel processing. Moreover, by utilizing an
Input Tensor Queue, we can accommodate temporary lapses in input images due to
device malfunctions or communication errors, enabling smooth streaming.

###  3.1 Batching the denoise step

In diffusion models, denoising steps are performed sequentially, resulting in
a proportional increase in the processing time of U-Net relative to the number
of steps. However, to generate high-fidelity images, it is necessary to
increase the number of steps. To resolve the high-latency generation in
interactive diffusion, we propose a method called Stream Batch.

The Stream Batch technique restructures sequential denoising operations into
batched processes, wherein each batch corresponds to a predetermined number of
denoising steps, as depicted in Fig. 2. The size of each batch is determined
by the number of these denoising steps. This approach allows for each batch
element to advance one step further in the denoising sequence via a single
pass through U-Net. By iteratively applying this method, it is possible to
effectively transform input images encoded at timestep t into their
corresponding image-to-image results at timestep t+n, thereby streamlining the
denoising procedure.

Stream Batch significantly reduces the need for multiple U-Net inferences. The
processing time does not escalate linearly with the number of steps. This
technique effectively shifts the trade-off from balancing processing time and
generation quality to balancing VRAM capacity and generation quality. With
adequate VRAM scaling, this method enables the production of high-quality
images within the span of a single U-Net processing cycle, effectively
overcoming the constraints imposed by increasing denoising steps.

###  3.2 Residual Classifier-Free Guidance

Classifier-free guidance (CFG) [7] is an algorithm that enhances the effect of
the original conditioning by performing vector calculations between an
unconditioning or a negative conditioning term [6] and the original
conditioning term (Eq. 1).

$\epsilon_{\tau_{i},\mathrm{cfg}}=\epsilon_{\tau_{i},\bar{c}}+\gamma(\epsilon_{\tau_{i},c}-\epsilon_{\tau_{i},\bar{c}}),$
(1)

where $\epsilon_{\tau_{i},c}$ and $\epsilon_{\tau_{i},\bar{c}}$ are predicted
residual noise at the denoising step $\tau_{i}$ with conditioning embedding
$c$ and negative conditioning embedding $\bar{c}$, respectively, and $\gamma$
is a guidance scale.

This allows for benefits such as strengthening the effect of the prompt.
However, to compute the negative conditioning residual noise, it is necessary
to pair each input latent variable with the negative conditioning embedding
and pass it through the U-Net at every inference time. To address this
drawback, we introduce Residual Classifier-Free Guidance (RCFG), which
significantly reduces the computational cost of the additional U-Net inference
for the negative conditioning embedding.

Firstly, the encoded input latent $x_{0}$ can be transferred to the noise
distribution $x_{\tau_{0}}$ as follows,

$x_{\tau_{0}}=\sqrt{\alpha_{\tau_{0}}}x_{0}+\sqrt{\beta_{\tau_{0}}}\epsilon_{0},$
(2)

where $\alpha_{\tau_{0}}$ and $\beta_{\tau_{0}}$ are values determined by a
noise scheduler and $\epsilon_{0}$ is a sampled noise from a Gaussian
$\mathcal{N}(0,I)$. Following the implementation of latent consistency model
[26], we can predict the data distribution
$\hat{x}_{0,{\tau_{i-1}},\mathrm{cfg}}$ and generate a next step noise
distribution $x_{\tau_{i},\mathrm{cfg}}$ using the CFG residual noise
$\epsilon_{{\tau_{i-1}},\mathrm{cfg}}$ as follows,

$\hat{x}_{0,{\tau_{i-1}},\mathrm{cfg}}=\frac{x_{{\tau_{i-1}},\mathrm{cfg}}-\sqrt{\beta_{\tau_{i-1}}}\epsilon_{{\tau_{i-1}},\mathrm{cfg}}}{\sqrt{\alpha_{\tau_{i-1}}}}$
(3)

$x_{{\tau_{i}},\mathrm{cfg}}=\sqrt{\alpha_{\tau_{i}}}\hat{x}_{0,{\tau_{i-1}},\mathrm{cfg}}+\sqrt{\beta_{\tau_{i}}}\epsilon_{i},$
(4)

Where ${\epsilon_{i}}\sim\mathcal{N}(0,I)$.

For the next step CFG calculation, instead of computing the next step negative
conditioning residual noise $\epsilon_{{\tau_{i}},\bar{c}}$, which requires an
additional U-Net computation cost, we hypothesize the existence of virtual
residual noise $\epsilon_{{\tau_{i}},\bar{c}^{\prime}}$, and a virtual
negative condition embedding $\bar{c}^{\prime}$ that predicts the original
input image’s latent $x_{0}$ from $x_{{\tau_{i}},\mathrm{cfg}}$, as shown in
Fig. 3. Similarly to Eq. 3, this virtual residual noise satisfies Eq. 5 to
predict the input latent $x_{0}$.

$x_{0}\approx\hat{x}_{0,{\tau_{i}},\bar{c}^{\prime}}=\frac{x_{{\tau_{i}},\mathrm{cfg}}-\sqrt{\beta_{\tau_{i}}}\epsilon_{{\tau_{i}},\bar{c}^{\prime}}}{\sqrt{\alpha_{\tau_{i}}}}$
(5)

Given the initial value $x_{0}$, and the subsequent values of
$x_{{\tau_{i}},\mathrm{cfg}}$ obtained sequentially through Eq. 4, the virtual
residual noise $\epsilon_{{\tau_{i}},\bar{c}^{\prime}}$ can be analytically
determined by employing these values with the Eq. 5:

$\epsilon_{{\tau_{i}},\bar{c}^{\prime}}=\frac{x_{{\tau_{i}},\mathrm{cfg}}-\sqrt{\alpha_{\tau_{i}}}x_{0}}{\sqrt{\beta_{\tau_{i}}}}$
(6)

With the virtual residual noise $\epsilon_{{\tau_{i}},\bar{c}^{\prime}}$
obtained from Eq. 6, we formulate RCFG as follows,

$\epsilon_{\tau_{i},\mathrm{cfg}}=\delta\epsilon_{\tau_{i},\bar{c}^{\prime}}+\gamma(\epsilon_{\tau_{i},c}-\delta\epsilon_{\tau_{i},\bar{c}^{\prime}})$
(7)

where $\delta$ is a magnitude moderation coefficient for the virtual residual
noise that softens the effect and the approximation error of the virtual
residual noise.

RCFG that uses the original input image latent $x_{0}$ as the residual term
can effectively generate results that diverge from the original input image
according to the magnitude of the guidance scale $\gamma$, thereby enhancing
the effect of conditioning without the need for additional U-Net computations.
We call this method Self-Negative RCFG.

Not only to deviate from the original input image $x_{0}$, but also to diverge
from any negative condition, we can compute the negative conditioning residual
noise $\epsilon_{\tau_{0},\bar{c}}$ using U-Net only onetime for the first
denoising step and use this to predict the negative conditioning denoised data
as follows,

$\hat{x}_{0,{\tau_{0}},\bar{c}}=\frac{x_{\tau_{0}}-\sqrt{\beta_{\tau_{0}}}\epsilon_{{\tau_{0}},\bar{c}}}{\sqrt{\alpha_{\tau_{0}}}}$
(8)

In Eq. 6, instead of $x_{0}$, using $\hat{x}_{0,{\tau_{0}},\bar{c}}$, we can
obtain the virtual residual noise $\epsilon_{{\tau_{i+1}},\bar{c}^{\prime}}$
that can effectively diverge the generation results from the controllable
negative conditioning embedding $\bar{c}$. We call this Onetime-Negative RCFG.

By combining Eq. 2, Eq. 6, and Eq. 8, both Self-Negative RCFG and Onetime-
Negative RCFG can be uniformly represented by the following equation,

$\epsilon_{{\tau_{i}},\bar{c}^{\prime}}=\frac{x_{{\tau_{i}},\mathrm{cfg}}-\sqrt{\alpha_{\tau_{i}}}x_{0}}{\sqrt{\beta_{\tau_{i}}}}+\sqrt{\frac{\alpha_{\tau_{i}}\beta_{\tau_{0}}}{\beta_{\tau_{i}}\alpha_{\tau_{0}}}}(\epsilon_{\tau_{0},\bar{c}}-\epsilon_{0})$
(9)

When $\epsilon_{\tau_{0},\bar{c}}=\epsilon_{0}$, Eq. 9 is identical to Eq. 6
and returns Self-Negative RCFG result.

In contrast to the conventional CFG, which requires $2n$ computations of
U-Net, the Self-Negative RCFG and Onetime-Negative RCFG necessitate only $n$
and $n+1$ computations of U-Net, respectively, where $n$ is the number of the
denoising steps.

Figure 5: Inference pipeline overview: The core diffusion inference pipeline,
including VAE and U-Net. By incorporating a denoising batch and pre-computed
prompt embedding cache, sampled noise cache, and scheduler values cache, the
inference pipeline is enhanced in terms of speed and enables real-time image
generation. Stochastic Similarity Filter (SSF) is designed to save GPU power
usage. It dynamically gates the pass of the diffusion model. This framework
realizes fast and energy-efficient real-time inference.

###  3.3 Input-Output Queue

The current bottleneck in high-speed image generation systems lies in the
neural network modules, including VAE and U-Net. To maximize the overall
system speed, processes such as pre-processing and post-processing of images,
which do not require handling by the neural network modules, are moved outside
of the pipeline and processed in parallel.

In the context of input image handling, specific operations, including
resizing of input images, conversion to tensor format, and normalization, are
meticulously executed. To address the disparity in processing frequencies
between the human inputs and the model throughput, we design an input-output
queuing system to enable efficient parallelization, as shown in Fig. 4. This
system operates as follows: processed input tensors are methodically queued
for Diffusion Models. During each frame, Diffusion Model retrieves the most
recent tensor from the input queue and forwards it to the VAE Encoder, thereby
triggering the image generation sequence. Correspondingly, tensor outputs from
the VAE Decoder are fed into an output queue. In the subsequent output image
handling phase, these tensors are subject to a series of post-processing steps
and conversion into the appropriate output format. Finally, the fully
processed image data is transmitted from the output handling system to the
rendering client.

###  3.4 Stochastic Similarity Filter

When images remain unchanged or show minimal changes, particularly in
scenarios without active user interaction or static environment, nearly
identical input images are often repeatedly fed into the VAE and U-Net. This
leads to the generation of identical or nearly identical images and
unnecessary consumption of GPU resources. In contexts involving continuous
inputs, such instances of unmodified input images can occasionally occur. To
tackle this issue and minimize unnecessary computational load, we propose a
strategy termed stochastic similarity filter (SSF), as shown in Fig. 5.

We calculate the cosine similarity between the current input Image $I_{t}$ and
the past reference frame Image $I_{\mathrm{ref}}$.

$S_{C}(I_{t},I_{\mathrm{ref}})=\frac{I_{t}\cdot
I_{\mathrm{ref}}}{\|I_{t}\|\|I_{\mathrm{ref}}\|}$ (10)

Based on this cosine similarity, we calculate the probability of skipping the
subsequent VAE and U-Net processes. It is given by

$\mathbf{P}(\mathrm{skip}|I_{t},I_{\mathrm{ref}})=\mathbf{max}\left\\{0,\>\frac{S_{C}(I_{t},I_{\mathrm{ref}})-\eta}{1-\eta}\right\\},$
(11)

where $\eta$ is the similarity threshold. This probability decides whether
subsequent processes like VAE Encoding, U-Net, and VAE Decoding should be
skipped or not. If not skipped, the input image at that time is saved and
updated as the reference image $I_{ref}$ for future use. This probabilistic
skipping mechanism allows the network to operate fully in dynamic scenes with
low inter-frame similarity, while in static scenes with high inter-frame
similarity, the network’s operational rate decreases, conserving computational
resources. The GPU usage is modulated seamlessly based on the similarity of
the input images, enabling smooth adaptation to scenes with varying dynamics.

Note: We emphasize that compared to determine whether we skip the compute via
a hard threshold, the proposed probability-sampling-based similarity filtering
strategy leads to a more smooth video generation. Because the hard threshold
is prone to making the video stuck which hurts the impression of watching
video streaming, while the sampling-based method significantly improves the
smoothness.

###  3.5 Pre-computation

The U-Net architecture requires both input latent variables and conditioning
embeddings. Typically, the conditioning embedding is derived from a prompt
embedding, which remains constant across different frames. To optimize this,
we pre-compute the prompt embedding and store it in a cache. In interactive or
streaming mode, this pre-computed prompt embedding cache is recalled. Within
U-Net, the Key and Value are computed based on this pre-computed prompt
embedding for each frame. We have modified the U-Net to store these Key and
Value pairs, allowing them to be reused. Whenever the input prompt is updated,
we recompute and update these Key and Value pairs inside U-Net.

For consistent input frames across different timesteps and to improve
computational efficiency, we pre-sample Gaussian noise for each denoising step
and store it in the cache. This ensures that while each denoising step has
distinct noise, every timestep retains the same noise ($x_{t,\tau}\neq
x_{t,\tau+1}$ and $x_{t+1,\tau}=x_{t,\tau}$). This approach is particularly
relevant for image-to-image tasks.

We also precompute $\alpha_{\tau}$ and $\beta_{\tau}$, the noise strength
coefficients for each denoising step $\tau$, defined as:

$x_{t}=\sqrt{\alpha_{\tau}}x_{0}+\sqrt{\beta_{\tau}}\epsilon$ (12)

This is a minor point in low throughput scenarios, but at frame rates higher
than 60 FPS, the overhead of recomputing these static values becomes
noticeable. If we achieve 60 FPS (i.e. 16.666 ms) with recomputing these
static values in the loop with pre-computed cache, we can achieve 62 FPS (i.e.
16.066 ms).

We note that we have a specific design of key-value cache for Latent
Consistency Models (LCM). As per the original paper, we need to compute
$c_{\mathrm{skip}}(\tau)$ and $c_{\mathrm{out}}(\tau)$ to satisfy the
following equation:

$f_{\theta}(x,\tau)=c_{\mathrm{skip}}(\tau)x+c_{\mathrm{out}}(\tau)F_{\theta}(x,\tau).$
(13)

The functions $c_{\mathrm{skip}}(\tau)$ and $c_{\mathrm{out}}(\tau)$ in
original LCM [17] is constructed as follows:

$c_{\mathrm{skip}}(\tau)=\frac{\sigma_{\mathrm{data}}^{2}}{(s\tau)^{2}+\sigma_{\mathrm{data}}^{2}},\quad
c_{\mathrm{out}}(\tau)=\frac{\sigma_{\mathrm{data}}s\tau}{\sqrt{\sigma_{\mathrm{data}}^{2}+(s\tau)^{2}}},$
(14)

where $\sigma_{\mathrm{data}}=0.5$, and the timestep scaling factor $s=10$. We
note that with $s=10$, $c_{\mathrm{skip}}(\tau)$ and $c_{\mathrm{out}}(\tau)$
approximate delta functions that enforce the boundary condition to the
consistency models. (i.e., at denoising step $\tau=0$,
$c_{\mathrm{skip}}(0)=1$, $c_{\mathrm{out}}(0)=0$; and at $\tau\neq 0$,
$c_{\mathrm{skip}}(\tau)=0$, $c_{\mathrm{out}}(\tau)=1$). At inference time,
there’s no need to recompute these functions repeatedly. We can either pre-
compute $c_{\mathrm{skip}}(\tau)$ and $c_{\mathrm{out}}(\tau)$ for all
denoising steps $\tau$ in advance or simply use constant values
$c_{\mathrm{skip}}=0$, $c_{\mathrm{out}}=1$ for any arbitrary denoising step
$\tau$.

###  3.6 Model Acceleration and Tiny AutoEncoder

We employ TensorRT to construct the U-Net and VAE engines, further
accelerating the inference speed. TensorRT is an optimization toolkit from
NVIDIA that facilitates high-performance deep learning inference. It achieves
this by performing several optimizations on neural networks, including layer
fusion, precision calibration, kernel auto-tuning, dynamic tensor memory, and
more. These optimizations are designed to increase throughput and efficiency
for deep learning applications.

To optimize speed, we configured the system to use static batch sizes and
fixed input dimensions (height and width). This approach ensures that the
computational graph and memory allocation are optimized for a specific input
size, leading to faster processing times. However, this means that if there is
a requirement to process images with different shapes (i.e., varying heights
and widths) or to use different batch sizes (including those for denoising
steps), a new engine tailored to these specific dimensions must be built. This
is because the optimizations and configurations applied in TensorRT are
specific to the initially defined dimensions and batch size, and changing
these parameters would necessitate a reconfiguration and re-optimization of
the network within TensorRT.

Besides, we employ a tiny AutoEncoder, which has been engineered as a
streamlined and efficient counterpart to the traditional Stable Diffusion
AutoEncoder [11, 20]. TAESD excels in rapidly converting latents into full-
size images and accomplishing decoding processes with significantly reduced
computational demands.

Figure 6: GPU Usage comparison under static scene. (GPU: RTX3060, Number of
frames: 20) The blue line represents the GPU usage with SSF, the orange line
indicates GPU usage without SSF, and the red line denotes the Skip probability
calculated based on the cosine similarity between input frames. Additionally,
the top of the plot displays input images corresponding to the same
timestamps. In this case, the character in the input images is only blinking.
Thus, this analysis compares GPU usage in a static scenario. Figure 7: GPU
Usage comparison under dynamic scene. (GPU: RTX4090, Number of frames: 1000)
The blue line represents the GPU usage with SSF, the orange line indicates GPU
usage without SSF, and the red line denotes the Skip probability calculated
based on the cosine similarity between input frames. Additionally, the top of
the plot displays input images corresponding to the same timestamps. In this
case, the character in the input images keeps moving dynamically. Thus, this
analysis compares GPU usage in a dynamic scenario.

##  4 Experiments

Table 1: Comparison of Average Inference Time at Different Denoising Steps with Speedup Factors. (GPU: NVIDIA RTX4090, CPU: Core i9-13900K, OS: Ubuntu22.04.3 LTS) Denoising Step | StreamDiffusion (ms) | StreamDiffusion w/o TRT (ms) | AutoPipeline Image2Image (ms)
---|---|---|---
1 | 10.65 (59.6x) | 21.34 (29.7x) | 634.40 (1x)
2 | 16.74 (39.3x) | 30.61 (21.3x) | 652.66 (1x)
4 | 26.93 (25.8x) | 48.15 (14.4x) | 695.20 (1x)
10 | 62.00 (13.0x) | 96.94 (8.3x) | 803.23 (1x)
Table 2: Comparison of Average Inference Time at Different Denoising Steps among Different CFG Methods Denoising Step | Self-Negative RCFG (ms) | Onetime-Negative RCFG (ms) | Normal CFG (ms)
---|---|---|---
1 | 11.04 (1.52x) | 16.55 (1.01x) | 16.74 (1x)
2 | 16.61 (1.64x) | 20.64 (1.32x) | 27.18 (1x)
3 | 20.64 (1.74x) | 27.25 (1.32x) | 35.91 (1x)
4 | 26.19 (1.90x) | 31.65 (1.57x) | 49.71 (1x)
5 | 31.47 (2.05x) | 36.04 (1.79x) | 64.64 (1x)
Figure 8: Average inference time comparison between Stream Batch and normal sequential denoising without TensorRT. Figure 9: Average inference time comparison between Stream Batch and normal sequential denoising with using TensorRT Table 3: The effect of different modules on average inference time at different denoising steps. The different module ablations are validated on image-to-image generation. Method | Denoising Step | Latency [ms]
---|---|---
Ours (txt2img) | 1 | 9.42
4 | 26.30
Ours (img2img) | 1 | 10.65
4 | 26.93
w/o stream batch | 1 | 10.70
4 | 36.54
w/o TRT | 1 | 21.34
4 | 48.15
w/o Pre-computation | 1 | 13.63
4 | 31.16
w/o TinyAutoEncoder | 1 | 49.72
4 | 65.48
w/o IO queue | 1 | 15.13
4 | 29.72
w/o any optimization | 1 | 76.78
4 | 134.37
with SSF | 1 | 10.66
4 | 27.02

We implement StreamDiffusion pipeline upon LCM, LCM-LoRA [17, 18] and SD-turbo
[21]. As a model accelerator, we use TensorRT and for the light weight
efficient VAE we use TAESD [11]. Our pipeline is compatible to the consumer
GPU. We test our pipeline on NVIDIA RTX4090 GPU, Intel Core i9-13900K CPU,
Ubuntu22.04.3 LTS, and NVIDIA RTX3060 GPU, Intel Core i7-12700K, Windows11 for
image generation. We note that we evaluate the throughput mainly via the
average inference time per image through processing 100 images.

###  4.1 Quantitative Evaluation of Denoising Batch

The efficiency comparison between the denoising batch and the original
sequential U-Net loop is shown in Fig. 8. When implementing a denoising batch
strategy, we observe a significant improvement in processing time. It achieves
a reduction by half when compared to a conventional U-Net loop at sequential
denoising steps. Even though applying TensorRT, the accelerate tool for neural
modules, our proposed Stream Batch still boosts the efficiency of original
sequential diffusion pipeline by a large margin at different denoising steps.

Figure 10: Results using no CFG, standard CFG, and RCFG with Self-Negative and
Onetime-Negative approaches. When compared to cases where CFG is not utilized,
the cases with CFG utilized can intensify the impact of prompts. In the
proposed method RCFG, a more pronounced influence of prompts was observed.
Both CFG and RCFG use guidance scale $\gamma=1.4$. For RCFG, the first two
rows use magnitude modelation coefficient $\delta=1.0$, and the third row uses
$\delta=0.5$. Figure 11: Text-to-Image generation results. We use four step
denoising for LCM-LoRA, and one step denoising for sd-turbo. Our
StreamDiffusion enables the real-time generation of images with quality
comparable to those produced using Diffusers AutoPipeline Text2Image.

Besides, we compare our method with the AutoPipelineForImage2Image, which is a
pipeline developed by Huggingface diffusers
111https://github.com/huggingface/diffusers. The average inference time
comparison is presented in Table. 1. Our pipeline demonstrates a substantial
speed increase. When we use TensorRT, StreamDiffusion achieves a minimum
speed-up 13.0 times when running the 10 denoising steps, and reaching up to
59.6 times in scenarios when involving a single denoising step. Even though
without TensorRT, StreamDiffusion achieves 29.7 times speed up compared to
AutoPipeline when using one step denoising, and 8.3 times speedup at 10 step
denoising.

Table. 2 presents a comparison of the inference times for StreamDiffusion
pipelines with RCFG and conventional CFG. The additional computations required
to apply Self-Negative RCFG are merely lightweight vector operations,
resulting in negligible changes in inference time compared to when Self-
Negative is not used. When employing Onetime-Negative RCFG, additional UNet
computations are necessary for the first step of the denoising process.
Therefore, One-time-negative RCFG and conventional CFG have almost identical
inference times for a single denoising step case. However, as the number of
denoising steps increases, the difference in inference time from conventional
CFG to both Self-Negative and Onetime-Negative RCFG becomes more pronounced.
At denoising step 5, a speed improvement of 2.05x is observed with Self-
Negative RCFG and 1.79x with Onetime-Negative RCFG, compared to conventional
CFG.

###  4.2 Energy Consumption

We then conduct a comprehensive evaluation of the energy consumption
associated with our proposed stochastic similarity filter (SSF), as depicted
in Figure. 6 and Figure. 7. These figures provide the GPU utilization patterns
when SSF (Threshold $\eta$ set at 0.98) is applied to input videos containing
scenes with periodic static characteristics. The comparative analysis reveals
that the incorporation of SSF significantly mitigates GPU usage in instances
where the input images are predominantly static and demonstrate a high degree
of similarity.

Figure. 6 delineates the results derived from a meticulously executed two-
denoise-step img2img experiment. This experiment was conducted on a 20-frame
video sequence, employing NVIDIA RTX3060 graphics processing units with or
without the integration of SSF. The experiment results indicate a substantial
decrease in average power consumption from 85.96w to 35.91w on one RTX3060
GPU. Using the same static scene input video with one NVIDIA RTX4090GPU, the
power consumption was reduced from 238.68w to 119.77w.

Furthermore, Figure. 7 expounds on the findings from a similar two-denoise-
step img2img experiment using one RTX4090GPU. This time the evaluation of
energy consumption is performed on a 1000-frame video featuring dynamic
scenes. Remarkably, even under drastically dynamic conditions, the SSF
efficiently extracted several frames exhibiting similarity from the dynamic
sequence. This process results in a noteworthy reduction in average power
consumption, from 236.13w to 199.38w. These findings underscore the efficacy
of the Stochastic Similarity Filter in enhancing energy efficiency,
particularly in scenarios involving static or minimally varying visual
content.

###  4.3 Ablation study

In our ablation study, as summarized in Table 3, we evaluate the average
inference time of our proposed method under various configurations to
understand the contribution of each component. Our proposed StreamDiffusion
achieves an average inference time of 10.98/9.42 ms and 26.93/26.30 ms for
denoising steps 1 and 4 on image-to-image/text-to-image generation,
respectively. When the stream batch processing is removed (’w/o stream
batch’), we observe a large time consumption increase, especially at 4
denoising steps. The absence of TensorRT (’w/o TRT’) leads to a more increase
in time cost. The removal of pre-computation also results in increased time
cost but not much. We attribute the reason to the limited number of key-value
computations in StableDiffusion. Besides, the exclusion of input-output queue
(’w/o IO queue’) also demonstrates an impact on average inference time, which
mainly aims to optimize the parallelization issue results from pre- and post-
processing. In the AutoPipelineImage2Image’s adding noise function, the
precision of tensors is converted from fp32 to fp16 for each request, leading
to a decrease in speed. In contrast, the StreamDiffusion pipeline standardizes
the precision of variables and computational devices beforehand. It does not
perform tensor precision conversion or computation device transfers during
inference. Consequently, even without any optimization (’w/o any
optimization’), our pipeline significantly outperforms the
AutoPipelineImage2Image in terms of speed. We also evaluate the impact on the
inference time of our pipeline with additional modules such as SSF and Self-
Negative RCFG. As shown in Table. 2 and Table. 3, these additional modules,
which only perform lightweight matrix and vector operations, do not have a
noticeable effect on inference time.

###  4.4 Qualitative Results

The generation results using the StreamDiffusion pipeline for real-time image-
to-image transformation are presented in Fig. 1. This pipeline enables image
generation with very low throughput from input images received in real time
from cameras or screen capture devices. At the same time, it is capable of
producing high quality images that effectively align to the specified prompt
conditions. These capabilities demonstrate the applicability of our pipeline
in various real time applications, such as real time game graphic rendering,
generative camera effect filters, real time face conversion, and AI-assisted
drawing.

The alignment of generated images to prompt conditioning using Residual
Classifier-Free Guidance (RCFG) is depicted in Fig. 10. The generated images,
without using any form of CFG, exhibit weak alignment to the prompt,
particularly in aspects like color changes or the addition of non-existent
elements, which are not effectively implemented. In contrast, the use of CFG
or RCFG enhances the ability to modify original images, such as changing hair
color, adding body patterns, and even incorporating objects like glasses.
Notably, the use of RCFG results in a stronger influence of the prompt
compared to standard CFG. RCFG, although limited to image-to-image
applications, can compute the vector for negative conditioning while
continuously referencing the latent value of the input image and the initially
sampled noise. This approach yields more consistent directions for the
negative conditioning vector compared to the standard CFG, which uses UNet at
every denoising step to calculate the negative conditioning vector.
Consequently, this leads to more pronounced changes from the original image.
However, there is a trade-off in terms of the stability of the generated
results. While Self-Negative RCFG enhances the prompt’s effectiveness, it also
has the drawback of increasing the contrast of the generated images. To
address this, adjusting the $delta$ in Eq. 7 can modulate the magnitude of the
virtual residual noise vector, thereby mitigating the rise in contrast.
Additionally, using Onetime-Negative RCFG with appropriately chosen negative
prompts can effectively mitigate contrast increases while improving prompt
adherence, as observed in Fig. 10. This approach allows the generated images
to blend more naturally with the original image.

Finally, the quality of standard text-to-image generation results is
demonstrated in Fig. 11. Using the sd-turbo model, high-quality images like
those shown in Fig. 11 can be generated in just one step. When images are
produced using our proposed StreamDiffusion pipeline and sd-turbo model in an
environment with GPU: RTX 4090, CPU: Core i9-13900K, and OS: Ubuntu 22.04.3
LTS, it’s feasible to generate such high-quality images at a rate exceeding
100fps. Furthermore, by increasing the batch size of images generated at once
to 12, our pipeline can continuously produce approximately 150 images per
second. The images enclosed in red frames shown Fig. 11 are generated in four
steps using community models merged with LCM-LoRA. While these LCM models
require more than 1 step for high quality image generation, resulting in a
reduction of speed to around 40fps, these LCM-LoRA based models offer the
flexibility of utilizing any base model, enabling the generation of images
with diverse expressions.

##  5 Conclusion

We propose StreamDiffusion, a pipeline-level solution for interactive
diffusion generation. StreamDiffusion consists of several optimization
strategies for both throughput and GPU usage, including stream batch, residual
classifier-free gudiance (RCFG), IO-queue for parallelization, stochastic
similarity filter, pre-computation, Tiny AutoEncoder and the use of model
acceleration tool. The synergistic combination of these elements results in a
marked improvement in efficiency. Specifically, StreamDiffusion achieves up to
91.07 frames per second (fps) on a standard consumer-grade GPU for image
generation tasks. This performance level is particularly beneficial for a
variety of applications, including but not limited to the Metaverse, online
video streaming, and broadcasting sectors. Furthermore, StreamDiffusion
demonstrates a significant reduction in GPU power consumption, achieving at
least a 1.99x decrease. This notable efficiency gain underscores
StreamDiffusion’s potential for commercial application, offering a compelling
solution for energy-conscious, high-performance computing environments.

##  6 Acknowledgments

We sincerely thank Taku Fujimoto and Huggingface team for their invaluable
feedback, courteous support, and insightful discussions.

## References

  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18208–18218, 2022.
  * Avrahami et al. [2023] Omri Avrahami, Ohad Fried, and Dani Lischinski.  Blended latent diffusion.  _ACM Trans. Graph._ , 42(4), 2023.
  * [3] James Betker, Gabriel Goh, Li Jing, TimBrooks, Jianfeng Wang, Linjie Li, LongOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu, YunxinJiao, and Aditya Ramesh.  Improving image generation with better captions.
  * Blattmann et al. [2023a] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv preprint arXiv:2311.15127_ , 2023a.
  * Blattmann et al. [2023b] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22563–22575, 2023b.
  * Du et al. [2020] Yilun Du, Shuang Li, and Igor Mordatch.  Compositional visual generation with energy based models.  In _Advances in Neural Information Processing Systems_ , pages 6637–6647. Curran Associates, Inc., 2020.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2023] Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu.  Tfmq-dm: Temporal feature maintenance quantization for diffusion models.  _arXiv preprint arXiv:2311.16503_ , 2023.
  * Jolicoeur-Martineau et al. [2021] Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and Ioannis Mitliagkas.  Gotta go fast when generating data with score-based models.  _arXiv preprint arXiv:2105.14080_ , 2021.
  * Kingma and Welling [2022] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2022.
  * Li et al. [2023] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer.  Q-diffusion: Quantizing diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 17535–17545, 2023.
  * Liu et al. [2023] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  _arXiv preprint arXiv:2309.06380_ , 2023.
  * Lu et al. [2022a] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022a.
  * Lu et al. [2022b] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Luo et al. [2023a] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023a.
  * Luo et al. [2023b] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao.  Lcm-lora: A universal stable-diffusion acceleration module.  _arXiv preprint arXiv:2311.05556_ , 2023b.
  * Luo et al. [2023c] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao.  Lcm-lora: A universal stable-diffusion acceleration module, 2023c.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. [2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  2022\.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho†, David Fleet, and Mohammad Norouzi.  Imagen: unprecedented photorealism × deep level of language understanding.  2022\.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023.
  * Watson et al. [2021] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan.  Learning to efficiently sample from diffusion probabilistic models.  _arXiv preprint arXiv:2106.03802_ , 2021.
  * Yin et al. [2023] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation.  _arXiv_ , 2023.
  * Zheng et al. [2023] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar.  Fast sampling of diffusion models via operator learning.  In _International Conference on Machine Learning_ , pages 42390–42402. PMLR, 2023.