ArtiFade: Learning to Generate High-quality Subject from Blemished Images
Shuya Yang Shaozhe Hao Yukang Cao Kwan-Yee K. Wong










ArtiFade
〈Φ〉





Orbital Support and Evolution of CX/OX Structures in Boxy/Peanut Bars
    Kathryne J. Daniel
    September 9, 2024
=====================================================================

§ ABSTRACT 

Subject-driven text-to-image generation has witnessed remarkable advancements in its ability to learn and capture characteristics of a subject using only a limited number of images. However, existing methods commonly rely on high-quality images for training and may struggle to generate reasonable images when the input images are blemished by artifacts. This is primarily attributed to the inadequate capability of current techniques in distinguishing subject-related features from disruptive artifacts. In this paper, we introduce ArtiFade to tackle this issue and successfully generate high-quality artifact-free images from blemished datasets. Specifically, ArtiFade exploits fine-tuning of a pre-trained text-to-image model, aiming to remove artifacts. The elimination of artifacts is achieved by utilizing a specialized dataset that encompasses both unblemished images and their corresponding blemished counterparts during fine-tuning. ArtiFade also ensures the preservation of the original generative capabilities inherent within the diffusion model, thereby enhancing the overall performance of subject-driven methods in generating high-quality and artifact-free images. We further devise evaluation benchmarks tailored for this task. Through extensive qualitative and quantitative experiments, we demonstrate the generalizability of ArtiFade in effective artifact removal under both in-distribution and out-of-distribution scenarios. 






§ INTRODUCTION



With the rapid advancement of generative diffusion models <cit.>, subject-driven text-to-image generation <cit.>, which aims to capture distinct characteristics of a subject by learning from a few images of the subject, has gained significant attention.
This approach empowers individuals to seamlessly incorporate their preferred subjects into diverse and visually captivating scenes by simply providing text conditions.
Representative works such as Textual Inversion <cit.> and DreamBooth <cit.> have shown promising results on this task. Specifically, Textual Inversion proposes to optimize a textual embedding to encode identity characteristics that provide rich subject information for subsequent generation. DreamBooth shares a similar idea but additionally fine-tunes the diffusion model to preserve more identity semantics. Plenty of successive efforts have been made to advance this task from various perspectives, including generation quality, compositionality, and efficiency <cit.>.

Both of the above mentioned methods, along with their follow-up works, however, rely heavily on the presence of unblemished input images that contain only relevant identity information. This is often expensive or even unavailable in real-world applications. 
Instead, in practical scenarios such as scraping web images of a desired subject, it is common to encounter images that are blemished by various visible artifacts such as watermarks, drawings, and stickers.
Additionally, there also exist invisible artifacts like adversarial noises <cit.> that are not easily detectable or removable using off-the-shelf tools. These artifacts can significantly impede the comprehensive learning of the subject and lead to a catastrophic decline in performance across multiple dimensions (see Fig. <ref>). This limitation arises from the feature confusion inherent in the existing subject-driven learning process. The process simultaneously captures subject-related features and disruptive artifact interference. It lacks the discriminative power to distinguish these two from each other, and fails to preserve the integrity of subject characteristics while mitigating negative effects caused by artifacts. 

As blemished inputs are inevitable in applications, a pressing challenge emerges: Can we effectively perform subject-driven text-to-image generation using blemished images? We term this novel problem (, generating subject-driven images from blemished inputs) as blemished subject-driven generation in this paper.

To answer the above question, we present ArtiFade, the first model to tackle blemished subject-driven generation by adapting vanilla subject-driven methods (, Textual Inversion <cit.> and DreamBooth <cit.>) to effectively extract subject-specific information from blemished training data. The key objective of ArtiFade is to learn the implicit relationship between natural images and their blemished counterparts through alignment optimization.
Specifically, we introduce a specialized dataset construction method to create pairs of unblemished images and their corresponding counterparts. These pairs can be applied to fine-tune various subject-driven approaches in the context of blemished subject-driven generation. Besides, we also observe fine-tuning an extra learnable embedding in the textual space, named artifact-free embedding, can enhance prompt fidelity in the blemished subject-driven generation.

We further introduce an evaluation benchmark that encompasses (1) multiple test sets of blemished images with diverse artifacts, and (2) tailored metrics for accurately assessing the performance of blemished subject-driven generation methods. A thorough experimental evaluation shows that our method consistently outperforms other existing methods, both qualitatively and quantitatively. Notably, exhibits superb capabilities in handling out-of-distribution (OOD) scenarios involving diverse types of artifacts that are distinct from the training data. This inherent generalizability indicates our model can effectively learn to discern and distinguish the patterns exhibited by artifacts and unblemished images, instead of overfitting to a specific type of artifacts.

In summary, our key contributions are as follows:

  
  * We are the first to tackle the novel challenge of blemished subject-driven generation. To address this task, we propose that fine-tunes diffusion models to align unblemished and blemished data.
  
  * We introduce an evaluation benchmark tailored for effectively assessing the performance of blemished subject-driven generation techniques.
  
  * We conduct extensive experiments and demonstrate that outperforms current methods significantly. We show noteworthy generalizability of , effectively addressing both in-distribution and out-of-distribution scenarios with various types of artifacts.



§ RELATED WORK







  
Text-to-image synthesis 
Text-to-image generation has attracted considerable attention in recent years by leveraging Generative Adversarial Networks (GANs) <cit.> and diffusion models <cit.>. Reed  reed2016generative was the first to integrate GANs into text-to-image generation. Since then, several influential works had been proposed <cit.>, demonstrating impressive results with improved resolution <cit.> and fidelity of fine details <cit.>. Diffusion models in text-to-image synthesis have also yielded remarkable results owing to their ability in generating precise and customized images that better align with individual text specifications <cit.>. 



  
Subject-driven generation 
Subject-driven generation has gained popularity due to its ability to generate personalized images based on a given set of subject images and text prompts. One prominent method in subject-driven generation is Textual Inversion <cit.>, which involves learning an embedding vector by minimizing the Latent Diffusion Model loss <cit.> on input images. The learned embedding vector can be effectively combined with text prompts, allowing seamless integration in the text-to-image generation process. Recent approaches <cit.> have significantly enhanced subject reconstruction fidelity by incorporating fine-tuning techniques.



  
Artifacts removal
Shadow and watermark removal are classic tasks in image processing and computer vision. At the early stage, most approaches for shadow removal or image recovery relied on the properties of intensity and illumination <cit.>. Some methods also incorporated color features to improve their results <cit.>. Deep learning techniques and Convolutional Neural Networks (CNNs) have played a significant role in advancing shadow removal methods and producing impressive results in recent years <cit.>. Several studies <cit.> have incorporated GANs to further enhance the results of shadow removal techniques. Moreover, with the increasing popularity of diffusion models in image generation, a novel diffusion-based method for shadow removal has recently been introduced <cit.>.



The most widely adopted methods for recovering concealed information from watermarked images include the application of generalized multi-image matting algorithms <cit.>, complemented by image inpainting techniques <cit.>, and the utilization of deep neural networks and CNNs <cit.>. Similar to shadow removal, GANs and Conditional GANs <cit.> are also widely used in watermark removal tasks <cit.>. Our work is closely related to these previously mentioned studies. We are the first to address the artifact issues in the realm of subject-driven text-to-image generation.


§ METHOD

 
Given a set of blemished input images, our objective is to eliminate their negative impacts on the quality of subject-driven image generation.
To achieve this goal, we present , an efficient framework that learns to discern and distinguish the patterns exhibited by various types of artifacts and unblemished images.
In this section, we focus exclusively on ArtiFade based on Textual Inversion. However, it is important to note that the ArtiFade framework can be generalized to other subject-driven generation methods.
As shown in Fig. <ref>, based on Textual Inversion incorporates two main components, namely the fine-tuning of the partial parameters (, key and value weights) in the diffusion model and the simultaneous optimization of an artifact-free embedding . We begin by discussing the preliminaries of the Latent Diffusion Model and Textual Inversion. In the following subsections, we elaborate our automatic construction of the training dataset, which consists of both blemished and unblemished data, illustrated in Sec. <ref>. 
We then introduce Artifact Rectification Training, a method for fine-tuning the model to accommodate blemished images, as discussed in Sec. <ref>.
We finally present the use of for handling blemished images in Sec. <ref>. 



  
Preliminary Latent Diffusion Model (LDM) <cit.> is a latent text-to-image diffusion model derived from Diffusion Denoising Probabilistic Model (DDPM) <cit.>. 
LDM leverages a pre-trained autoencoder to map image features between the image and latent space. This autoencoder comprises an encoder ℰ, which transforms images into latent representations, and a decoder 𝒟, which converts latent representations back into images. The autoencoder is optimized using a set of images so that the reconstructed image x̂≈𝒟(ℰ(x)). Additionally, LDM introduces cross-attention layers <cit.> within the U-Net <cit.>, enabling the integration of text prompts as conditional information during the image generation process.
The LDM loss is defined as

    ℒ_LDM := 𝔼_z ∼ℰ(ℐ), y, ϵ∼ N(0,1)[‖ϵ - ϵ_θ(z_t, t, y) ‖_2^2],
 
where ℰ encodes the image ℐ into the latent representation z. Here, z_t denotes the noisy latent representation at timestep t, ϵ_θ refers to the denoising network, and y represents the text condition that is passed to the cross-attention layer.

Based on LDM, Textual Inversion <cit.> aims to capture the characteristics of a specific subject from a small set of images. Specifically, Textual Inversion learns a unique textual embedding by minimizing Eq. (<ref>) on a few images that contain the particular subject. It can produce promising generation results with high-quality inputs, but fails on input images that are blemished by artifacts (see Fig. <ref>). This problem arises from the inherent limitation of Textual Inversion in learning shared characteristics exhibited in the input images without the capability in differentiating artifacts from unblemished subjects. In this paper, we aim to address this issue on deteriorated generation quality of Textual Inversion in the presence of blemished images.




 §.§ Dataset Construction


Existing subject-driven generation methods operate under the assumption of unblemished training data, consisting of solely high-quality images devoid of any artifacts. However, this assumption does not align with real-world applications, where obtaining blemished images from the internet is a commonplace. To address this blemished subject-driven generation in this paper, we first construct a training set that incorporates both unblemished images and their blemished counterparts that are augmented with artifacts.



  
Augmentation of multiple artifacts 
We construct our dataset by collecting a multi-subject set 𝒞 of N image subsets from existing works <cit.> and a set ℬ of L different artifacts:

    𝒞 = {𝒮_i}_i=1^N,   𝒮_i = {ℐ_i,j}_j=1^M_i,   ℬ={β_k}_k=1^L,

where 𝒮_i denotes the image subset corresponding to the ith subject, M_i is the total number of images in 𝒮_i, and β_k represents a type of artifact for image augmentation. Our dataset 𝒟 can then be constructed by applying each artifact β_k to each image ℐ in 𝒮_i separately, i.e.,

    𝒮_i^β_k = {ℐ_i,j^β_k}_j=1^M_i,   𝒟 = {𝒮_i, {𝒮_i^β_k}_k=1^L}_i=1^N,

where ℐ_i,j^β_k is the counterpart of ℐ_i,j augmented with the specific artifact β_k. Some examples of original images and their augmented versions with distinct artifacts can be found in Fig. <ref>. See the Appendix for more visualizations.




  
Blemished textual embedding 

For each blemished subset, we perform Textual Inversion to optimize a blemished textual embedding [𝚅_i^β_k] 
, i.e.,

    𝒮_i^β_k  Textual Inversion   [𝚅_i^β_k],
       i=1,2,...,N;  k=1,2,...,L

By applying Eq. (<ref>) on N subsets with L types of artifacts, we end up with a set of N × L blemished textual embeddings 𝒱 = {[𝚅_i^β_k]}_i=1,k=1^N,L, which will be used in the subsequent model fine-tuning. As we have illustrated in Fig. <ref>, directly prompting the diffusion model with [𝚅_i^β_k] will lead to a significant decrease in generation quality. Consequently, our objective is to robustly handle blemished embeddings and effectively eliminate the detrimental impact of artifacts. We achieve this by devising a partial fine-tuning paradigm for the pre-trained diffusion model on the constructed training set 𝒟, as elaborated in the following subsection.



 §.§ Artifact Rectification Training
 

After establishing the curated dataset 𝒟, we embark on training a generalizable framework on 𝒟, capable of generating unblemished images using blemished textual embeddings.
To this end, we propose artifact rectification training, which consists of two key components, namely partial fine-tuning of a pre-trained diffusion model and the optimization of an artifact-free embedding, to eliminate the artifacts and distortions in the generated images.

We fine-tune only partial parameters that are involved in processing the textual conditions. This strategy allows us to optimize the relevant components associated with the blemished textual embedding [𝚅_i^β_k]. Considering that only the key and value weights in the diffusion model's cross-attention layer are involved in the processing of textual embedding, we choose to fine-tune these two types of parameters W^k and W^v. Moreover, we find that optimizing an additional embedding, , in the textual space with partial parameters could improve prompt fidelity by retaining the textual information of the model, as presented later in Sec. <ref>.



  
Training objective
During each iteration, we will first randomly sample an unblemished image ℐ_i,j from the training set 𝒟 and a type of artifact β_k ∈ℬ to obtain

the blemished textual embedding [𝚅_i^β_k] ∈𝒱 that is optimized on the blemished subset 𝒮_i^β_k.

Specifically, given the sampled blemished textual embedding [𝚅_i^β_k], we form the prompt “a photo of [𝚅_i^β_k]”, which will be input to the text encoder to acquire the text condition y_i^β_k. 
Our optimization objective will then be defined as reconstructing the unblemished image ℐ_i,j by conditioning the denoising process on the text condition y_i^β_k.
Thus, we can formulate the final loss for training as 

    ℒ_ArtiFade :=   𝔼_z ∼ℰ(ℐ_i,j), y_i^β_k, ϵ∼ N(0,1)
       [‖ϵ - ϵ_{W^k, W^v, ⟨Φ⟩}(z_t, t, y_i^β_k)‖_2^2],

where {W^k, W^v, ⟨Φ⟩} is the set of the trainable parameters of .



 §.§ Subject-driven Generation with Blemished Images



After artifact rectification training, we obtain the ArtiFade model, prepared for the task of blemished subject-driven generation.
Given a test image set 𝒮^β^
'_test in which all images are blemished by an arbitrary artifact β^',

the model can generate high-quality subject-driven images using blemished samples with ease.

Specifically, we first obtain the blemished textual embedding [𝚅_𝚝𝚎𝚜𝚝^β^'] by applying Textual Inversion on the test set 𝒮^β^
'_test. We then simply infer the model with a given text prompt that includes the blemished textual embedding, , “a photo of [𝚅_𝚝𝚎𝚜𝚝^β^']”. At the operational level, the sole distinction between our approach and vanilla Textual Inversion lies in inputting text prompts containing [𝚅_𝚝𝚎𝚜𝚝^β^'] into the fine-tuned instead of the pre-trained diffusion model. This simple yet effective method resolves the issue of Textual Inversion's incapacity to handle blemished input images, bearing practical utility.



  
Details of ArtiFade models 

We choose N= 20 subjects, including pets, plants, containers, toys, and wearable items to ensure a diverse range of categories.
We experiment with the ArtiFade model based on Textual Inversion trained with visible watermark artifacts, namely .
The training set of involves L_WM= 10 types of watermarks, characterized by various fonts, orientations, colors, sizes, and text contents. Therefore, we obtain 200 blemished subsets in total within the training set of . We fine-tune for a total of 16k steps.





§ EXPERIMENT




 §.§ Implementation Details

We employ the pre-trained LDM <cit.> following the official implementation of Textual Inversion <cit.> as our base diffusion model. We train the blemished textual embeddings for 5k steps using Textual Inversion.
We use a learning rate of 5e-3 to optimize our Artifact-free embedding and 3e-5 for the partial fine-tuning of key and value weights. Note that all other parameters within the pre-trained diffusion model remain frozen. All experiments are conducted on 2 NVIDIA RTX 3090 GPUs. In the main paper, we focus on the comparison with Textual Inversion and DreamBooth to demonstrate the efficiency of our proposed contributions. See the Appendix for additional comparisons and applications.



 §.§ Evaluation Benchmark




  
Test dataset 
We construct the test dataset using 16 novel subjects that differ from the subjects in the training set. These subjects encompass a wide range of categories, including pets, plants, toys, transportation, furniture, and wearable items. We form the visible test artifacts into two categories: (1) in-distribution watermarks () containing the same type as the training data, and (2) out-of-distribution watermarks () of different types from the training data.

Within the  and , we synthesize 5 distinct artifacts for each category, resulting in 80 test sets. 





  
Evaluation metrics 
We evaluate the performance of blemished subject-driven generation from three perspectives: (1) the fidelity of subject reconstruction, (2) the fidelity of text conditioning, and (3) the effectiveness of mitigating the negative impacts of artifacts. Following common practice <cit.>, we use CLIP <cit.> and DINO <cit.> similarities for measuring these metrics. For the first metric, we calculate the CLIP and DINO similarity between the generated images and the unblemished version of the input images, respectively denoted as ICLIP and IDINO. For the second metric, we calculate the CLIP similarity between the generated images and the text prompt, denoted as TCLIP. For the third metric, we calculate the relative ratio of similarities between generated images and unblemished input images compared to their blemished versions, defined as

    R^CLIP = I^CLIP / I^CLIP_β  R^DINO = I^DINO / I^DINO_β

where I^CLIP_β and I^DINO_β respectively denote CLIP and DINO similarities between the generated images and the blemished input images. A relative ratio greater than 1 indicates that generated images resemble unblemished images more than blemished counterparts, suggesting fewer artifacts. Conversely, a ratio less than 1 indicates that generated images are heavily distorted with more artifacts. We use DINO ViT-S/16 <cit.> and CLIP ViT-B/32 <cit.> to compute all metrics.



 §.§ Quantitative Comparisons
 
We conduct both in-distribution and out-of-distribution quantitative evaluations of our method and compare it to Textual Inversion with blemished embeddings. We additionally report the results using Textual Inversion on unblemished images as a reference, although it is not a direct comparison to our model.



  
In-distribution (ID) analysis We consider the in-distribution scenarios by testing on .


In Tab. <ref>, we can observe that the use of blemished embeddings in Textual Inversion leads to comprehensive performance decline including: (1) lower subject reconstruction fidelity (, IDINO and ICLIP) due to the subject distortion in image generation; (2) lower efficiency for artifact removal (, RDINO and RCLIP) due to inability to remove artifacts; (3) lower prompt fidelity (, TCLIP) since the prompt-guided background is unrecognizable due to blemishing artifacts.
In contrast, our method consistently achieves higher scores than Textual Inversion with blemished embeddings across the board, demonstrating the efficiency of in various aspects.



  
Out-of-distribution (OOD) analysis We pleasantly discover that possesses the capability to handle out-of-distribution scenarios, owing to its training with watermarks of diverse types.
We consider the out-of-distribution (OOD) scenarios for by testing it on ,

as presented in Tab. <ref>. Similar to ID evaluation, all of our metrics yield higher results than Textual Inversion with blemished embeddings. These results further demonstrate the generalizability of our method.



 §.§ Qualitative Comparisons
  
We present qualitative comparisons between the output generated via ArtiFade and Textual Inversion with blemished textual embeddings, including in-distribution scenarios in Fig. <ref> and out-of-distribution scenarios in Fig. <ref>. 


  
In-distribution analysis
The images generated by Textual Inversion exhibit noticeable limitations when using blemished textual embeddings. Specifically, as depicted in Fig. <ref>, all rows predominantly exhibit cases of incorrect backgrounds that are highly polluted by watermarks. By using ArtiFade, we are able to eliminate the background watermarks.



  
Out-of-distribution analysis 
In addition, we conduct experiments with our to showcase its capability to remove out-of-distribution watermarks, as shown in Fig. <ref>. It is important to note that in the first row, the watermark in the input images may not be easily noticed by human eyes upon initial inspection due to the small font size and high image resolution. However, these artifacts have a significant effect when used to train blemished embeddings for generating images. ArtiFade effectively eliminates the artifacts on the generated images, improving reconstruction fidelity and background accuracy, hence leading to substantial enhancements in overall visual quality.




 §.§ ArtiFade with DreamBooth



The ArtiFade fine-tuning framework is not limited to Textual Inversion with textual embedding; it can also be generalized to DreamBooth. We use the same training dataset and blemished subsets as in the case of the (, N= 20, L_WM= 10). The vanilla DreamBooth fine-tunes the whole UNet model, which conflicts with the fine-tuning parameters of ArtiFade. We therefore use DreamBooth with low-rank approximation (LoRA)[<https://huggingface.co/docs/peft/main/en/task_guides/dreambooth_lora>] to train LoRA adapters <cit.> for the text encoder, value, and query weights of the diffusion model for each blemished subset using Stable Diffusion v1-5. For simplicity, we will use DreamBooth to refer to DreamBooth with LoRA below. During the fine-tuning of DreamBooth-based ArtiFade, we load the pre-trained adapters and only unfreeze key weights since value weights are reserved for DreamBooth subject information. In Tab. <ref>, it is evident that our method, based on DreamBooth, yields the highest scores among all cases. Our method also maintains DreamBooth's advantages in generating images with higher subject fidelity and more accurate text prompting, outperforming ArtiFade with Textual Inversion. We show some qualitative results in Fig. <ref>.




 §.§ Invisible Artifacts Blemished Subject Generation



ArtiFade demonstrates exceptional performance in handling subjects characterized by intricate features and blemished by imperceptible artifacts. We collect 20 human figure datasets from the VGGFace2 dataset <cit.>. We then use the Anti-DreamBooth <cit.> ASPL method to add adversarial noises to each group of images, producing 20 blemished datasets for fine-tuning a DreamBooth-based ArtiFade model. The model is fine-tuned for 12k steps. As illustrated in Fig. <ref>, our approach surpasses the DreamBooth in differentiating the learning of adversarial noises from human face features. In contrast to DreamBooth, which is fooled into overfitting adversarial noises, thereby generating images with a heavily polluted background, our model reconstructs human figures in image generation while maintaining high fidelity through text prompting.




 §.§ Ablation Studies
 

We conduct ablation studies to demonstrate the efficiency of our method by comparing with three alternative variants, which encompass (1) 𝚅𝚊𝚛_𝙰, where we solely fine-tune the artifact-free embedding; (2) 𝚅𝚊𝚛_𝙱, where we fine-tune parameters related to image features, , query weights W^q, along with the artifact-free embedding, and (3) 𝚅𝚊𝚛_𝙲, where we fine-tune key and value weights, , W^k and W^v, exclusively. We use our  to compare it with other variants by testing on the .



  
Effect of partial fine-tuning As shown in Tab. <ref>, compared to 𝚅𝚊𝚛_𝙰, our full method yields higher scores on all metrics by a significant margin, except for RDINO. This is reasonable, as the artifact-free embedding can be easily overfitted to the training data, resulting in generated images that resemble a fusion of training images (Fig. <ref>, 𝚅𝚊𝚛_𝙰). As a result, the denominator of RDINO, namely the similarity between the generated image and the blemished image, is significantly decreased, leading to a high RDINO. Due to similar reason, 𝚅𝚊𝚛_𝙰 shows lowest IDINO, ICLIP, and TCLIP among all variants, indicating that it fails to reconstruct the correct subject.  
Overall, both quantitative and qualitative evaluation showcases that solely optimizing the artifact-free embedding is insufficient to capture the distinct characteristics presented in the blemished input image, demonstrating the necessity of partial fine-tuning.



  
Effect of fine-tuning key and value weights As shown in Tab. <ref> and Fig. <ref>, 𝚅𝚊𝚛_𝙱 yields unsatisfactory outcomes in all aspects compared to ours. The lower RDINO and RCLIP suggest that the generated images retain artifact-like features and bear closer resemblances to the blemished subsets. Furthermore, the reduced TCLIP indicates diminished prompt fidelity, as the approach fails to accurately reconstruct the subject from the blemished embeddings, which is also evidenced by Fig. <ref>. These findings suggest that fine-tuning the parameters associated with text features yields superior enhancements in terms of artifact removal and prompt fidelity.




  
Effect of the artifact-free embedding With 𝚅𝚊𝚛_𝙲, we exclude the optimization of artifact-free embedding. In Tab. <ref>, we can observe that 𝚅𝚊𝚛_𝙲 yields higher IDINO and ICLIP but lower RDINO and RCLIP compared to our , which indicates that the approach achieves higher subject fidelity but lower efficiency in eliminating artifacts when generating images. Since our primary objective is to generate artifact-free images from blemished textual embedding, our chooses to trade off subject reconstruction fidelity for the ability to remove artifacts. Additionally, this approach produces lower TCLIP than ours, suggesting that the artifact-free embedding effectively improves the model's capability to better preserve text information (see Fig. <ref>).




§ MORE APPLICATIONS

We apply our to more artifact cases, such as stickers and glass effects, showcasing its broad applicability.


  
Sticker removal. In Fig. <ref>, we test on input images that are blemished by cartoon stickers.
The cartoon sticker exhibits randomized dimensions and is positioned arbitrarily within each image.
can effectively eliminate any stickers while concurrently addressing improper stylistic issues encountered during image generation.


  
Glass effect removal. 
We further test on input images that are blemished by glass effect in Fig. <ref>.
We apply a fluted glass effect to images to replicate real-life scenarios where individuals capture photographs of subjects positioned behind fluted glass. This glass can have specific reflections and blurring, which may compromise the overall quality of image generation when using Textual Inversion. The use of our model can fix the distortions of the subjects and the unexpected background problem, significantly improving image quality.


§ CONCLUSION


In conclusion, we introduce  to address the novel problem of generating high-quality and artifact-free images in the blemished subject-driven generation. Our approach involves fine-tuning a diffusion model along with artifact-free embedding to learn the alignment between unblemished images and blemished information.
We present an evaluation benchmark to thoroughly assess a model's capability in the task of blemished subject-driven generation. We demonstrate the effectiveness of ArtiFade in removing artifacts and addressing distortions in subject reconstruction under both in-distribution and out-of-distribution scenarios. 






























































































































































































































































































































































































































Appendix
    Kathryne J. Daniel
    September 9, 2024
======================





§ TRAINING DATASET DETAILS

Our training dataset consists of 20 training subjects, used for the fine-tuning stage of our models. We show an example image of each subject in Fig. <ref>. In Fig. <ref>, we showcase several unblemished images alongside their corresponding blemished versions, each featuring one of the 10 watermark types.





§ TEST DATASET DETAILS

In Fig. <ref>, we illustrate our  watermark types (see the first row) and  watermark types (see the second row). The  watermarks are chosen from the training watermarks displayed in Fig. <ref>. On the other hand, the  watermarks differ in font size, orientation, content, or color from all the training watermarks presented in Fig. <ref>.




§ ANALYSIS OF WATERMARK DENSITY

In Fig. <ref>, we present results to illustrate the impact of varying watermark densities (, varying qualities), highlighting the robust ability of our to remove watermarks under all conditions.





§ ANALYSIS OF UNBLEMISHED IMAGE RATIO

We employ our to evaluate the performance when the input images contain different proportions of unblemished images. We test our and Textual Inversion on five ratios of unblemished images: 100%, 75%, 50%, 25%, and 0%. The results are shown in Fig. <ref>. 

Notably, even when there is only one blemished image in the second column example, the impact on Textual Inversion is already evident, which deteriorates as the ratio decreases. Instead, our method effectively eliminates artifacts in all settings of unblemished image ratio, demonstrating its versatility in real-life scenarios.





§ ANALYSIS OF TRAINING DATASET SIZE

We conduct an analysis to investigate the impact of the number of training subjects (, the size of the training dataset) on the performance of our model. We utilize the same set of artifacts L_WM = 10, as described in Method in the main paper. We construct blemished training datasets in four different sizes: (1) with 5 subjects, (2) with 10 subjects, (3) with 15 subjects, and (4) with 20 subjects. We generate 50, 100, 150, and 200 blemished datasets for each of these cases. Subsequently, we fine-tune four distinct ArtiFade models, each with 16k training steps.

We compare the models trained using different data sizes under the in-distribution scenario (see Fig. <ref>) and under the out-of-distribution scenario (see Fig. <ref>). We note that when the number of training subjects is less than 15, IDINO and TCLIP are relatively lower than the other two cases in both ID and OOD scenarios. This observation can be attributed to a significant likelihood of subject or background overfitting during the reconstruction and image synthesis processes, as visually illustrated in Fig. <ref> and Fig. <ref>. However, as the number of training subjects reaches or exceeds 15, we observe a convergence in the values of IDINO and TCLIP, indicating a reduction in subject overfitting. Regarding RDINO, we note that all cases exhibit values greater than one, with a slightly increasing trend as the number of training subjects rises. 






§ FAILURE CASES

We present several failure cases when applying ArtiFade based on Textual Inversion. We demonstrate the limitations of our in Fig. <ref>. Despite the model's ability to eliminate watermarks, we still encounter issues with incorrect subject color, as shown in Fig. <ref>, which arises due to the influence of the watermark color. We also encounter incorrect subject identity in some cases, as demonstrated in Fig. <ref>. One possible reason is that the watermarks significantly contaminate the images, causing the learning process of embedding to focus on the contaminated visual appearance instead of the intact subject. 

Another failure case is subject overfitting, as shown in Fig. <ref>. In this case, the constructed subject overfits with a similar subject type that appears in the training dataset. This problem occurs because the blemished embedding of the testing subject closely resembles some blemished embeddings of the training subjects. Surprisingly, we find those problems can be solved by using ArtiFade based on DreamBooth, which is mentioned in Sec. 4.5. Therefore, we recommend using ArtiFade based on DreamBooth when encountering the limitations mentioned above.





§ ADDITIONAL COMPARISON WITH TEXTUAL INVERSION

We use the same training subjects with N=20 from Sec. 3.3 to train an ArtiFade model named using red circle artifacts. For the training set of , due to the simplicity of red circles, we only synthesize a single blemished subset (, L_RC= 1) for each subject, deriving 20 blemished subsets in total. We augment each image with a red circle mark that is randomly scaled and positioned on the source image. Considering the small scale of 's datasets, we only fine-tune for 8k steps. 
We further introduce , which applies only one type of artifact (, red circle) to our 16 test subjects, resulting in 16 test sets. We test both and on . The quantitative and qualitative results are shown in Tab. <ref> and Fig. <ref>, respectively.



  
Quantitative results analysis.
From Tab. <ref>, we can observe that both and yield higher results in nearly all cases than Textual Inversion <cit.> with blemished inputs, showing the capability of our models to eliminate artifacts and generate subjects with higher fidelity. It is important to note that the  is considered out-of-distribution with respect to . Nevertheless, the metrics produced by remain comparable to those of , with a minor difference observed. These results provide additional evidence supporting the generalizability of our .



  
Qualitative results analysis. As illustrated in Fig. <ref>, Textual Inversion struggles with accurate color reconstruction. It also showcases subject distortions and introduces red-circle-like artifacts during image generation when using blemished embeddings. In contrast, our (see Fig. <ref>) and (see Fig. <ref>) are capable of generating high-quality images that accurately reconstruct the color and identities of subjects without any interference from artifacts during the image synthesis.







§ ADDITIONAL QUALITATIVE COMPARISONS

We present additional qualitative results comparing our ArtiFade models with Textual Inversion <cit.> and DreamBooth <cit.> in Fig. <ref>. We employ and ArtiFade based on DreamBooth mentioned in Sec. 4.5.
Textual Inversion generates images with distorted subjects and backgrounds contaminated by watermarks, whereas DreamBooth can effectively capture intricate subject details and accurately reproduce watermark patterns. In contrast, our models (, TI-based and DB-based ArtiFade) generate images devoid of watermark pollution with correct subject identities for both in-distribution (see the first three rows in Fig. <ref>) and out-of-distribution (see the last two rows in Fig. <ref>) cases. Notably, our method based on DreamBooth preserves the high fidelity and finer detail reconstruction benefits of vanilla DreamBooth, even in the context of blemished subject-driven generation.

In Fig. <ref>, we show qualitative results for subjects with complex features (e.g., human faces) using our models, Textual Inversion, DreamBooth and Break-a-Scene <cit.>. Break-a-Scene can separate multiple subjects inside one image. We use Break-a-scene to generate human-only images. However, we find that Break-a-scene fails to separate humans from artifacts, resulting in polluted images. 
As a result, our methods (, TI-based and DB-based ArtiFade) consistently surpass Textual Inversion, DreamBooth, and Break-a-Scene, achieving high-quality image generation of complex data in in-distribution cases, as shown in the first two rows of Fig. <ref>, and out-of-distribution cases, as illustrated in the last row of Fig. <ref>.



§ MORE APPLICATIONS

We explore more applications of our , demonstrating its versatility beyond watermark removal. As shown in Fig. <ref>, our model exhibits the capability to effectively eliminate unwanted artifacts from images, enhancing their visual quality. Furthermore, our model showcases the ability to recover incorrect image styles induced by artifacts, thereby restoring the intended style of the images.



§ SOCIAL IMPACT

Our research addresses the emerging challenge of generating content from images with embedded watermarks, a scenario we term blemished subject-driven generation. Users often source images from the internet, some of which may contain watermarks intended to protect the original author's copyright and identity. However, our method is capable of removing various types of watermarks, potentially compromising the authorship and copyright protection. This could lead to increased instances of image piracy and the generation of illicit content. Hence, we advocate for legal compliance and the implementation of usage restrictions to govern the deployment of our technique and subsequent models in the future. 




