DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture
Qianlong Xiang Miao Zhang Yuzhang Shang Jianlong Wu Yan Yan Liqiang Nie



[
    [
    September 9, 2024
=====================



§ ABSTRACT 
Diffusion models (DMs) have demonstrated exceptional generative capabilities across various areas, while they are hindered by slow inference speeds and high computational demands during deployment.
The most common way to accelerate DMs involves reducing the number of denoising steps during generation, achieved through faster sampling solvers or knowledge distillation (KD).
In contrast to prior approaches, we propose a novel method that transfers the capability of large pretrained DMs to faster architectures.
Specifically, we employ KD in a distinct manner to compress DMs by distilling their generative ability into more rapid variants.
Furthermore, considering that the source data is either unaccessible or too enormous to store for current generative models, we introduce a new paradigm for their distillation without source data, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM).
Generally, our established DKDM framework comprises two main components: 1) a DKDM objective that uses synthetic denoising data produced by pretrained DMs to optimize faster DMs without source data, and 2) a dynamic iterative distillation method that flexibly organizes the synthesis of denoising data, preventing it from slowing down the optimization process as the generation is slow.
To our knowledge, this is the first attempt at using KD to distill DMs into any architecture in a data-free manner.
Importantly, our DKDM is orthogonal to most existing acceleration methods, such as denoising step reduction, quantization and pruning.
Experiments show that our DKDM is capable of deriving 2× faster DMs with performance remaining on par with the baseline.
Notably, our DKDM enables pretrained DMs to function as “datasets” for training new DMs.




§ INTRODUCTION


The advent of Diffusion Models (DMs) <cit.> heralds a new era in the generative modeling domain, garnering widespread acclaim for their exceptional capability in producing samples of remarkable quality <cit.>.
These models have rapidly ascended to a pivotal role across a spectrum of generative applications, notably in the fields of image, video and audio <cit.>.
However, the generation via those models is significantly slow because the sampling process involves iterative noise estimation over thousands of time steps, which poses a challenge for practical deployment, particularly for consumer devices <cit.>.

To accelerate diffusion models, as illustrated in Figure <ref> (b)&(c), existing methods can be categorized into two pathways: reducing denoising steps and speeding up inference process of denoising networks for each step.
Compared with the standard generation of DMs as shown in Figure <ref> (a), existing methods of the first category <cit.> focus on reducing denoising steps of the lengthy sampling process.
The second category focuses on reducing the inference time of each denoising step, through quantization <cit.>, pruning <cit.>, and so on.
However, these studies often overlook the efficiency in denoising network architecture, a critical factor in the generation speed of DMs, which can be improved by efficient architecture design, i.e., Neural Architecture Search <cit.>.



Knowledge Distillation (KD) <cit.> is an effective method for transferring the capabilities of large, cumbersome models to smaller ones, either with similar <cit.> or different architectures <cit.>.
The conventional KD methods typically require simultaneous access to the dataset to sample the training data, which is then used to align the student model's behavior with that of the teacher model <cit.>.
However, the vast data requirements of training generative deep neural networks, such as GPT-4 <cit.> and Stable Diffusion,[<https://github.com/runwayml/stable-diffusion>] present significant challenges for traditional KD methods, as they often necessitate direct dataset access, which complicates data storage and accessibility.
Additionally, while weights of such models are frequently released, the corresponding datasets may remain confidential due to privacy concerns.
Recent studies have explored using generative models to eliminate the demand for the source data in KD <cit.>.
Motivated by these studies, in this paper, we explore a novel KD paradigm that distills generative ability of DMs without source data, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM).

The DKDM paradigm hinges on addressing two critical challenges.
The first challenge involves optimizing a student model through the synthetic denoising data, instead of the source data.
The second challenge involves flexibly organizing the synthesis of denoising data, preventing it from becoming the main bottleneck in slowing the optimization process, as generation of DMs is inherently slow.
For the former, the optimization objective used in traditional DMs, as described by <cit.>, is inappropriate due to the absence of the data.
To address this, we have especially designed a DKDM objective that aligns closely with the original DM optimization objective.
For the latter challenge, the most straightforward approach is to utilize the teacher DMs to generate a comprehensive dataset, matching the size of the source dataset employed for training the teacher.
This dataset is then used to train the student model following the standard training algorithm <cit.>.
However, this method becomes impractical for extremely large datasets, like those utilized by models such as Stable Diffusion, due to excessive computational and storage demands.
To overcome this, we introduce a dynamic iterative distillation method that efficiently collects denoising data with varying noise levels, rather than generating real-like samples.
This method significantly reduces computation and storage requirements.
Importantly, our approach is complementary to most previously established methods, as summarized in Figure <ref> (b)&(c).
Our experiment results validate that our DKDM is able to derive 2× faster DMs while still able generate high-quality samples.
Additionally, our method allows pretrained DMs to act as “dataset” for training new DMs, thereby reducing the storage demands for further research on DMs.









§ PRELIMINARIES ON DIFFUSION MODELS


In diffusion models <cit.>, a Markov chain is defined to add noises to data and then diffusion models learn the reverse process to generate data from noises.

Forward Process. Given a sample ^0 ∼ q(^0) from the data distribution, the forward process iteratively adds Gaussian noise for T diffusion steps[We set T to 1,000 for all our experiments.] with the predefined noise schedule (β_1, …, β_T):

    q(^t | ^t-1)   =𝒩(^t ; √(1-β_t)^t-1, β_t I), 
    
        q(^1: T | ^0)   =∏_t=1^T q(^t | ^t-1),

until a completely noise ^T ∼𝒩(0, I) is obtained. According to <cit.>,
adding noise t times sequentially to the original sample ^0 to generate a noisy sample ^t can be simplified to a one-step calculation as follows:

    q(^t | ^0)=𝒩(^t ; √(α̅_t)^0,(1-α̅_t) I),


    ^t=√(α̅_t)^0+√(1-α̅_t)ϵ,

where α_t:=1-β_t, α̅_t:=∏_s=0^t α_s and ϵ∼𝒩(0, I).

Reverse Process. The posterior q(^t-1 | ^t) depends on the data distribution, which is tractable conditioned on ^0:

    q(^t-1 | ^t, ^0)=𝒩(^t-1 ; μ̃(^t, ^0), β̃_t I),

where μ̃_t(^t, ^0) and β̃_t can be calculated by:

    β̃_t:=1-α̅_t-1/1-α̅_tβ_t,


    μ̃_t(^t, ^0):=√(α̅_t-1)β_t/1-α̅_t^0+√(α_t)(1-α̅_t-1)/1-α̅_t^t.

Since ^0 in the data is not accessible during generation, a neural network parameterized by θ is used for approximation:

    p_θ(^t-1 | ^t)=𝒩(^t-1 ; μ_θ(^t, t), Σ_θ(^t, t) I).

Optimization. To optimize this network, the variational bound on negative log likelihood 𝔼[-log p_θ] is estimated by:

    L_vlb=𝔼_^0,ϵ,t[D_KL(q(^t-1|^t,^0)||p_θ(^t-1|^t)].

<cit.> found that predicting ϵ is a more efficient way when parameterizing μ_θ(^t,t) in practice, which can be derived by Equation (<ref>) and Equation (<ref>):

    μ_θ(^t, t)=1/√(α_t)(^t-β_t/√(1-α̅_t)ϵ_θ(^t, t)).


Thus, a reweighted loss function is designed as the objective to optimize L_vlb:

    L_simple=𝔼_^0, ϵ, t[ϵ-ϵ_θ(^t, t)^2].


Improvement. In original DDPMs, L_simple offers no signal for learning Σ_θ(^t,t) and <cit.> fixed it to β_t or β̃_t. <cit.> found it to be sub-optimal and proposed to parameterize Σ_θ(^t, t) as a neural network whose output v is interpolated as:

    Σ_θ(^t, t)=exp(v logβ_t+(1-v) logβ̃_t).


To optimize Σ_θ(^t, t), <cit.> use L_vlb, in which a stop-gradient is applied to the μ_θ(^t,t) because it is optimized by L_simple. The final hybrid objective is defined as:

    L_hybrid=L_simple + λ L_vlb,

where λ is used for balance between the two objectives.[We set λ to 1 for all our experiments.] Guided by (<ref>), the process of training and sampling are shown in Algorithm <ref> and Algorithm <ref> in Appendix <ref>.



§ DATA-FREE KNOWLEDGE DISTILLATION FOR DIFFUSION MODELS



In this section, we introduce a novel paradigm, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM).
Section <ref> details the DKDM paradigm, focusing on two principal challenges: the formulation of the optimization objective and the acquisition of denoising data for distillation.
Section <ref> describes our proposed optimization objective tailored for DKDM.
Section <ref> details our proposed method for effective collection of denoising data.



 §.§ DKDM Paradigm


The DKDM paradigm represents a novel data-free KD approach for DMs.
Unlike conventional KD methods, DKDM aims to leverages KD to transfer the generative capabilities of DMs to models with any architecture, while eliminating the need for access to large or proprietary datasets.
This approach poses two primary challenges: 1) optimizing DMs through synthetic denoising data instead of source data, and 2) devising methods to flexibly collect denoising data for KD as the generation is slow.

r0.5
  

  
    < g r a p h i c s >

  font=small
  Illustration of DKDM Paradigm. (a): standard training of DMs. (b): an intuitive baseline. (c): our proposed framework.
  
  



In standard training of DMs, as depicted in Figure <ref> (a),
a training sample ^0 ∼𝒟 is selected along with a timestep t ∼[1, 1000] and random noise ϵ∼𝒩(0, I).
The input ^t is computed using Equation (<ref>), and the denoising network is optimized according to Equation (<ref>) to generate outputs close to ϵ.
However, without dataset access, DKDM cannot obtain training data (^t, t, ϵ) to employ this standard method.
A straightforward approach for DKDM, termed the intuitive baseline and depicted in Figure <ref> (b), involves using DMs pretrained on 𝒟 to generate a synthetic dataset 𝒟^',[The number of samples in the synthetic dataset 𝒟^' is equal to those in original dataset 𝒟] which is then used to train new DMs with varying architectures.
Despite its simplicity, creating 𝒟^' is time-intensive and impractical for large datasets.

We propose an effective framework for DKDM paradigm, outlined in Figure <ref> (c), which incorporates a DKDM Objective (described in Section <ref>) and a strategy for collecting denoising data ℬ_i during optimization (detailed in Section <ref>).
This framework addresses the challenges of distillation without source dataset and reduces the costs associated with the intuitive baseline, since the synthetic ℬ_i requires much less computation than 𝒟^'.



 §.§ DKDM Objective


Given a dataset 𝒟, the original optimization objective for a diffusion model with parameters θ involves minimizing the KL divergence 𝔼_^0,ϵ,t[D_KL(q(^t-1|^t,^0)p_θ(^t-1|^t))].
Our proposed DKDM objective encompasses two primary goals: (1) eliminating the diffusion posterior q(^t-1|^t,^0) and (2) removing the diffusion prior ^0 ∼ q(^t | ^0) from the KL divergence, since they both are dependent on ^0 from the dataset 𝒟.

Eliminating the diffusion posterior q(^t-1|^t,^0).
In our framework, we introduce a teacher DM with parameters , trained on dataset 𝒟.
This model can generate samples that conform to the learned distribution 𝒟^'.
Optimized with the objective (<ref>), the distribution 𝒟^' within a well-learned teacher model closely matches 𝒟.
Our goal is for a student DM, parameterized by , to replicate 𝒟^' instead of 𝒟^', thereby obviating the need for q during optimization.

Specifically, the pretrained teacher model is optimized via the hybrid objective in Equation (<ref>), which indicates that both the KL divergence D_KL(q(^t-1|^t,^0)p_(^t-1|^t)) and the mean squared error 𝔼_^t, ϵ, t[ϵ-ϵ_(^t, t)^2] are minimized.
Given the similarity in distribution between the teacher model and the dataset, we propose a DKDM objective that optimizes the student model through minimizing D_KL(p_(^t-1|^t))p_(^t-1|^t))) and 𝔼_^t[ϵ_(^t, t)-ϵ_(^t, t)^2].
Indirectly, the DKDM objective facilitates the minimization of D_KL(q(^t-1|^t,^0)p_(^t-1|^t))) and 𝔼_^0, ϵ, t[ϵ-ϵ_(^t, t)^2], despite the inaccessibility of the posterior.
Consequently, we propose the DKDM objective as follows:

    L_DKDM=L_simple^'+λ L_vlb^',

where L_simple^' guides the learning of μ_ and L_vlb^' optimizes Σ_, as defined in following equations:

    L_simple^' = 𝔼_^0,ϵ,t[ϵ_(^t,t)-ϵ_(^t,t)^2],


    L_vlb^' = 𝔼_^0,ϵ,t[D_KL(p_(^t-1|^t)p_(^t-1|^t)],


where q(^t-1|^t,^0) is eliminated whereas the term ^t ∼ q(^t | ^0) remains to be removed.

Removing the diffusion prior q(^t | ^0).
Considering the generative ability of the teacher model, we utilize it to generate ^t as a substitute for ^t ∼ q(^t | ^0).
We define a reverse diffusion step ^t-1∼ p_(^t-1|^t) through the equation ^t-1=g_(^t, t).
Next,we represent a sequence of t reverse diffusion steps starting from T as G_(t).
Note that G_(0)=ϵ where ϵ∼𝒩(0, I).
For instance, G_(2) yields ^T-2=g_(g_(ϵ, T), T-1).
Consequently, ^t is obtained by ^t=G_(T-t) and the objective L_simple^' and L_vlb^' are reformulated as follows:

    L_simple^' = 𝔼_^t, t[ϵ_(^t,t)-ϵ_(^t,t)^2],


    L_vlb^' = 𝔼_^t,t[D_KL(p_(^t-1|^t)p_(^t-1|^t)].


By this formulation, the necessity of ^0 in L_DKDM is removed by naturally leveraging the generative ability of the teacher model.
Optimized by the proposed L_DKDM, the student progressively learns the entire reverse diffusion process from the teacher model without reliance on the source datasets.

However, the removal of the diffusion posterior and prior in the DKDM objective introduces a significant bottleneck, resulting in notably slow learning rates.
As depicted in Figure <ref> (a), standard training for DMs enable straightforward acquisition of noisy samples _i^t_i at an arbitrary diffusion step t ∼ [1,T] using Equation (<ref>).
These samples are compiled into a training data batch ℬ_j={_i^t_i}, with j representing the training iteration.
Conversely, our DKDM objective requires obtaining a noisy sample _i^t=G_(T-t_i) through T-t_i denoising steps.
Consequently, considering the denoising steps as the primary computational expense, the worst-case time complexity of assembling a denoising data batch _j={_i^t_i} for distillation is 𝒪(Tb), where b denotes the batch size.
This complexity significantly hinders the optimization process.
To address this issue, we introduce a method called dynamic iterative distillation, detailed in Section <ref>.



 §.§ Efficient Collection of Denoising Data


In this section, we present our efficient strategy for gathering denoising data for distillation, illustrated in Figure <ref>.
We begin by introducing a basic iterative distillation method that allows the student model to learn from the teacher model at each denoising step, instead of requiring the teacher to denoise multiple times within every training iteration to create a batch of noisy samples for the student to learn once.
Subsequently, to enhance the diversity of noise levels within the batch samples, we develop an advanced method termed shuffled iterative distillation, which allows the student to learn denoising patterns across varying time steps.
Lastly, we refine our approach to dynamic iterative distillation, significantly augmenting the diversity of data in the denoising batch. This adaptation ensures that the student model acquires knowledge from a broader array of samples over time, avoiding repetitive learning from identical samples.

Iterative Distillation.
We introduce a method called iterative distillation, which closely aligns the optimization process with the generation procedure.
In this approach, the teacher model consistently denoises, while the student model continuously learns from this denoising.
Each output from the teacher's denoising step is incorporated into some batch for optimization, ensuring the student model learns from every output.
Specifically, during each training iteration, the teacher performs g_(^t,t), which is a single-step denoising, instead of G_(t), which would involve t-step denoising.
Initially, a batch _1={_i^T} is formed from a set of sampled noises _i^T∼𝒩(0,I).
After one step of distillation, the batch _2={_i^T-1} is used for training.
This process is iterated until _T={_i^1} is reached, indicating that the batch has nearly become real samples with no noise.
The cycle then restarts with the resampling of noise to form a new batch _T+1={_i^T}.
This method allows the teacher model to provide an endless stream of data for distillation.
To further improve the diversity of the synthetic batch _j={_i^t_i}, we investigate from the perspectives of noise level t_i and sample _i.

Shuffled Iterative Distillation.
Unlike standard training, the t values in an iterative distillation batch remain the same and do not follow a uniform distribution, resulting in significant instability during distillation.
To mitigate this issue, we integrated a method termed shuffle denoise into our iterative distillation.
Initially, a batch _0^s={_i^T} is sampled from a Gaussian distribution.
Subsequently, each sample undergoes random denoising steps, resulting in _1^s={_i^t_i}, with t_i following a uniform distribution.
This batch, _1^s, initiates the iterative distillation process.
By ensuring diversity in the t_i values within the batch, this method balances the impact of different t values during distillation.



Dynamic Iterative Distillation.
There is a notable distinction between standard training and iterative distillation regarding the flexibility in batch composition.
Consider two samples, _1 and _2, within a batch without differentiating their noise level.
During standard training, the pairing of _1 and _2 is entirely random.
Conversely, in iterative distillation, batches containing _1 almost always include _2.
This departure from the principle of independent and identically distributed samples in a batch can potentially diminish the model's generalization ability.

r0.5


0.5






To better align the distribution of the denoising data with that of the standard training batch, we propose a method named dynamic iterative distillation.
As shown in Figure <ref>
, this method employs shuffle denoise to construct an enlarged batch set _1^+={_i^t_i}, where size |_j^+|=ρ T|_j^s|, where ρ is a scaling factor.
During distillation, a subset _j^s is sampled from _j^+ through random selection for optimization.
The one-step denoised samples replace their counterparts in _j+1^+.
This method only has a time complexity of 𝒪(b) and significantly improves distillation performance.
The final DKDM objective is defined as:

    L_DKDM^⋆=L_simple^⋆+λ L_vlb^⋆,


    L_simple^⋆ = 𝔼_(^t,t) ∼^+[ϵ_(^t,t)-ϵ_(^t,t)^2],


    L_vlb^⋆ = 𝔼_(^t,t) ∼^+[D_KL(p_(^t-1|^t)p_(^t-1|^t)],

where ^t and t is produced by our proposed dynamic iterative distillation. The complete algorithm is detailed in Algorithm <ref>.







§ EXPERIMENTS


This section details extensive experiments that demonstrate the effectiveness of our proposed DKDM.
In Section <ref>, we establish appropriate metrics and baselines for evaluation.
Section <ref> compares the performance of baselines and our DKDM with different architectures.
Additionally, we show that our DKDM can be combined with other methods to accelerate DMs.
Finally, Section <ref> describes an ablation study that validates the effectiveness of our proposed dynamic iterative distillation.



 §.§ Experiment Setting


Datasets and teacher diffusion models.
Our DKDM paradigm inherently eliminates the necessity for datasets.
However, the pretrained teacher models employed are trained on specific datasets.
We utilize three distinct pretrained DMs as teacher models, following the configurations introduced by <cit.>, and these models are all based on convolutional architecture.
These models have been pretrained on CIFAR10 at a resolution of 32 × 32 <cit.>, CelebA at 64 × 64 <cit.> and ImageNet at 32 × 32 <cit.>.

Metrics.
The distance between the generated samples
and the reference samples can be estimated by the Fréchet Inception Distance (FID) score <cit.>.
In our experiments, we utilize the FID score as the primary metric for evaluation.
Additionally, we report sFID <cit.>, Inception Score (IS) <cit.> as secondary metrics.
Following previous work <cit.>, we generate 50K samples for DMs and we use the full training set in the corresponding dataset to compute the metrics.
Without additional contextual states, all the samples are generated through 50 Improved DDPM sampling steps <cit.> and the speed is measured by the average time taken to generate 256 images on a single NVIDIA A100 GPU.
All of our metrics are calculated by ADM TensorFlow evaluation suite <cit.>.

Baseline.
As the DKDM is a new paradigm proposed in this paper, previous methods are not suitable to serve as baselines.
Therefore, in the data-free scenario, we take the intuitive baseline depicted in Figure <ref> (a) as the baseline.
Specifically, the teacher model consumes a lot of time to generate a substantial number of high-quality samples, equivalent in quantity to the source dataset, through 1,000 DDPM denoising steps.
These samples then serve as the synthetic dataset (𝒟^') for the training of randomly initialized student models, following the standard training (Algorithm <ref> in Appendix <ref>).
We use the performance obtained from this method as our baseline for comparative analysis.

r.45


.45






 §.§ Main Results


Effectiveness.
Table <ref> shows the performance comparison of our DKDM and baselines.
Our DKDM consistently outperforms the baselines, demonstrating superior generative quality when maintaining identical architectures for the derived DMs.
This performance validates the efficacy of our proposed DKDM objective and dynamic iterative distillation approach.
The improvement over baselines is attributed to the complexity of the reverse diffusion process, which baselines struggle to learn, whereas the knowledge from pretrained teacher models is easier to learn, highlighting the advantage of our DKDM.
Additionally, DKDM facilitates the distillation of generative capabilities into faster and more compact models, as evidenced by the 2 × faster architectures evaluated.
The parameter count and generative speed are detailed in Table <ref>, with further information on hyperparameters and network architecture available in Appendix <ref>.
Appendix <ref> includes some exemplary generated samples, illustrating that the student DMs, derived through DKDM, are capable of producing high-quality images.
Nevertheless, a limitation noted in Table <ref> is that the performance of these student DMs falls behind their teacher counterparts, which will be discussed further in Section <ref>.

We further evaluated the performance of these student models across a diverse range of architectures.
Specifically, we tested five different model sizes by directly specifying the architecture, bypassing complex methods like neural architecture search.
Both the teacher and student models employ Convolutional Neural Networks (CNNs) and the results are shown in Figure <ref>fig:compression.
Detailed descriptions of these architectures are available in Appendix<ref>.
Typically, we distilled a 14M model from a 57M teacher model, maintaining competitive performance and doubling the generation speed.
Additionally, the 44M and 33M student models demonstrated similar speeds, suggesting that DKDM could benefit from integration with efficient architectural design techniques to further enhance the speed and quality of DMs.
This aspect, however, is beyond our current scope and is designated for future research.

r.35


.35






Cross-Architecture Distillation.
Our DKDM transcends specific model architectures, enabling the distillation of generative capabilities from CNN-based DMs to Vision Transformers (ViT) and vice versa.
We utilized DiT <cit.> for ViT-based DMs to further affirm the superiority of our approach.
Detailed structural descriptions are available in Appendix <ref>.
For experimental purposes, we pretrained a small ViT-based DM to serve as the teacher.
As shown in Table <ref>, DKDM effectively facilitates cross-architecture distillation, yielding superior performance compared to baselines.
Additionally, our results suggest that CNNs are more effective as compressed DMs than ViTs.

Combination with Orthogonal Methods.
The DMs derived by our DKDM are compatible with various orthogonal methods, such as denoising step reduction, quantization, pruning and so on.
Here we conducted experiments to integrate the DDIM method, which reduces denoising steps, as illustrated in Figure <ref> fig:ddim.
This integration demonstrates that DDIM can further accelerate our derived student models, although with some performance trade-offs.



 §.§ Ablation Study


To validate our designed paradigm DKDM, we tested FID score of our progressively designed methods, including iterative, shuffled iterative, and dynamic iterative distillation, over 200K training iterations.
For the dynamic iterative distillation, the parameter ρ was set to 0.4.
The results, shown in Figure <ref> fig:strategy, demonstrate that our dynamic iterative distillation strategy not only converges more rapidly but also delivers superior performance.
The convergence curve for our method closely matches that of the baseline, which confirms the effectiveness of the DKDM objective in alignment with the standard optimization objective (<ref>).

Further experiments explored the effects of varying ρ on the performance of dynamic iterative distillation.
As illustrated in Figure <ref> fig:rho, higher ρ values enhance the distillation process up to a point, beyond which performance gains diminish.
This outcome supports our hypothesis that dynamic iterative distillation enhances batch construction flexibility, thereby improving distillation efficiency.
Beyond a certain level of flexibility, increasing ρ does not significantly benefit the distillation process. Further stability analysis of our dynamic iterative distillation is available in Appendix <ref>.




§ DISCUSSION AND FUTURE WORK



r.45
.45



    







The primary concept of our proposed DKDM paradigm is illustrated in Figure <ref>.
In this paradigm, the teacher DM , is trained on a real dataset 𝒟, which follows the distribution 𝒟^'.
There are two critical relationships between 𝒟 and 𝒟^'.
First, the distribution 𝒟^' of a well-trained teacher closely approximates 𝒟.
Second, the FID scores, when computed using 𝒟 as a reference, correlate with those using 𝒟^', as demonstrated by the linear fitting in Figure <ref>.
This correlation underpins the effectiveness of the DKDM paradigm.
By transferring the distribution 𝒟^' from the teacher DM to a lighter student DM, DKDM enables the student to generate data whose distribution closely approximates 𝒟.

However, in practice, there is invariably some discrepancy between 𝒟 and 𝒟^', limiting the performance of the student model.
We report these scores, denoted as FID^' and sFID^', calculated over the distribution 𝒟^' instead of 𝒟 in Table <ref>.
The results indicate that the FID^' and sFID^' scores of the student closely mirror those of the teacher, suggesting effective optimization.
Nevertheless, these scores are inferior to those of the teacher, primarily due to the gap between 𝒟 and 𝒟^'.
A potential solution to enhance DKDM involves improving the generative capabilities of the teacher, which we leave as a direction for future work.



§ CONCLUSIONS

In this paper, we introduce DKDM, a novel paradigm designed to efficiently distill the generative abilities of pretrained diffusion models into more compact and faster models with any architecture.
Our experiments demonstrate the effectiveness of DKDM across three datasets, showcasing its ability to compress models to various sizes and architectures.
A key advantage of our method is its ability to perform efficient distillation without direct data access, significantly facilitating ongoing research and development in the field.
Moreover, our DKDM is compatible with most existing methods for accelerating diffusion models and can be integrated with them.



plainnat




§ TRAINING AND SAMPLING OF DDPMS



In this section, we present the algorithms for training and sampling from standard DDPMs. The specific details have been previously introduced in Section <ref>.







§ HYPERPARAMETERS





 §.§ Hyperparameters for DKDM



For experiments of both DKDM and baseline, we use the hyperparameters specified by <cit.>, which are also in line with those adopted by <cit.>, as reported in Table <ref>.
Settings of our training process are basically the same with <cit.> and <cit.>, including mixed precision training, EMA and so on. All the models are trained on 8 NVIDIA A100 GPUs (with 40G memory).






 §.§ Hyperparameters for Model Compression


Table <ref> shows the architectures of different students trained in Section <ref> for model compression.





 §.§ Hyperparameters for Cross-Architecture Distillation


Similar to the configuration used by <cit.>, the hyperparameters employed for the ViT-based diffusion model in Section <ref> are presented in Table <ref>. This particular configuration was chosen due to the characteristic of ViT-based diffusion models generally requiring more time for image generation compared to their CNN-based counterparts. To illustrate this point, when generating 2,500 images on single A100 40GB GPU with 50 Improved DDPM steps, it takes approximately 57 seconds for a 57M CNN diffusion model, whereas a 19M ViT diffusion model requires 66 seconds. Our final choice of this configuration was driven by the aim to achieve fairness in our experimentation and analysis.





§ EXEMPLARY GENERATED SAMPLES


In this section, we demonstrate some exepmlary generated samples of our derived 2×-speed and 1/4-size student DMs.





§ ANALYSIS: RANDOM DISCARD



During our exploration, we discovered that the utilization of the Random Discard technique proves to be a straightforward yet highly effective approach for enhancing the distillation process. The idea behind it involves the random elimination of some batch of noisy samples generated by the teacher model during the iterative distillation.
For instance, in iterative distillation during the initial five training iterations, batches _1, _3, _4 may be discarded, while _2,_5 are utilized for the student's learning.
We present an analysis of the impact of random discarding in our devised methodologies. Specifically, we introduce the parameter p to denote the probability of discarding certain noisy samples. Subsequently, we apply varying discard probabilities to the iterative distillation, shuffled iterative distillation, and dynamic iterative distillation, and assess their respective performance alterations over a training duration of 200k iterations.
The outcomes are presented in Figure <ref>. It is noteworthy that both iterative distillation and shuffled iterative distillation face limitations in constructing flexible batches, where random discard emerges as a noteworthy solution to enhance their efficacy. Conversely, for dynamic iterative distillation, when ρ attains a sufficiently large value, it becomes apparent that random discard fails to confer additional advantages. This observation underscores the inherent stability of our dynamic iterative distillation method and we ultimately omitted random discard from the final implementation. This is beneficial because of its inefficiency in requiring the teacher model to prepare a larger number of noisy samples.







§ SOCIAL IMPACT


As an effective method for compressing and accelerating diffusion models, we believe that DKDM has the potential to significantly reduce the deployment costs associated with these models, thereby facilitating more widespread use of diffusion models for generating desired content.
However, it is imperative to acknowledge that, as generative models, diffusion models, while offering creative applications across various scenarios, may also engender consequences such as the production of dangerous or biased content.

Our DKDM is capable of mimicking the generative capabilities of a wide array of existing diffusion models without accessing the source datasets, which leads to derived models inheriting the flaws and limitations of these pre-existing models.
For instance, if the training data of a pre-trained diffusion model contains sensitive or personal information collected without explicit consent, derived models may still risk leaking this data.
Consequently, the potential societal harms of our approach primarily hinge on the negative impacts brought about by the existing diffusion models themselves.
Addressing how to mitigate the adverse effects inherent in diffusion models remains a critical area of research.



