iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptation
Hayeon Jo Hyesong Choi Minhee Cho Dongbo Min


Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
    Tornike Karchkhadze1 Mohammad Rasool Izadi2 Ke Chen1 Gerard Assayag3Shlomo Dubnov1
    Accepted Sep 3 2024 to ApJ Letters

======================================================================================



§ ABSTRACT 
Transfer learning based on full fine-tuning (FFT) of the pre-trained encoder and task-specific decoder becomes increasingly complex as deep models grow exponentially. Parameter efficient fine-tuning (PEFT) approaches using adapters consisting of small learnable layers have emerged as an alternative to FFT, achieving comparable performance while maintaining high training efficiency.
However, the inflexibility of the adapter with respect to input instances limits its capability of learning task-specific information in diverse downstream tasks. In this paper, we propose a novel PEFT approach, input-Conditioned transFormer, termed iConFormer, that leverages a dynamic adapter conditioned on the input instances. To secure flexible learning ability on input instances in various downstream tasks, we introduce an input-Conditioned Network (iCoN) in the dynamic adapter that enables instance-level feature transformation. To be specific, iCoN generates channel-wise convolutional kernels for each feature and transform it 
using adaptive convolution process to effectively capture task-specific and fine-grained details tailor to downstream tasks. Experimental results demonstrate that by tuning just 1.6% to 2.8% of the Transformer backbone parameters, iConFormer achieves performance comparable to FFT in monocular depth estimation and semantic segmentation, while outperforming it in image classification and instance segmentation. Also, the proposed method consistently outperforms recent PEFT methods for all the tasks mentioned above.



















§ INTRODUCTION



As deep neural networks (DNNs) grow increasingly complex, transfer learning—fine-tuning pre-trained models with task-specific data for downstream tasks—has become a widely adopted solution across diverse applications, including image classification, semantic segmentation, and object detection, to name a few. For instance, the model consisting of the pre-trained encoder <cit.> and task-specific decoder is fine-tuned, achieving remarkable performance gain when compared to the model trained from scratch. 
However, training large, complex models separately for each task and dataset is highly inefficient.
Recently, parameter efficient fine-tuning (PEFT) approaches <cit.> that maximize the efficiency in terms of training parameters have emerged as an alternative to the above-mentioned full fine tuning (FFT) methodologies, achieving competitive performance even with limited computing resources while simplifying the training processes and deployment.



This remarkable progress in vision tasks 
 is primarily driven by approaches including prompt tuning <cit.> and adapter <cit.>, which have been successfully applied in natural language processing (NLP) tasks. Visual prompt tuning <cit.> is the first study to explore the potential of prompt tuning in visual recognition tasks, laying the foundation for the prompt tuning in the field of computer vision. In addition, the adapter-based PEFT methods <cit.> achieve significant training efficiency by applying the adapter used in NLP to the Vision Transformer (ViT) and its variants <cit.>.



While most PEFT-based approaches yield comparable performance to baseline methods using the FFT in the image classification task, they do not yet provide sufficiently satisfactory performance to compete with the FFT in other complex downstream tasks. 
Additionally, the scalability of prompt-based methods <cit.> is significantly limited, leading to considerable performance degradation as the number of learnable parameters increases, as reported in <cit.>.
Adapter-based approaches <cit.> incorporate lightweight modules to reduce the number of trainable parameters while maintaining stable performance across a range of trainable parameter scales. However, the adapters always apply the same transformation to input features, ignoring individual characteristics of input instances. This may not be an issue when fully tuning DNNs, but it could be a limiting factor in improving performance in adapter-based PEFT methods. Namely, the inflexibility with respect to input instances exacerbates the transfer capability of adapter-based models with small learnable parts to downstream tasks, limiting their ability to capture unique and task-specific information. 

Additionally, the Vision Transformer (ViT) backbone <cit.> used in adapter-based models tends to focus on global information rather than fine-grained local details within an image. While this limitation can be partially addressed by employing the Swin Transformer <cit.>, which utilizes local attention mechanisms, the constraint on the number of learnable parameters in PEFT still restricts the Swin Transformer's capability to effectively capture local features. Consequently, this negatively affects the performance in dense prediction tasks that require fine-grained information.









To address the aforementioned issues, we propose a novel PEFT approach, input-Conditioned transFormer, termed iConFormer, that leverages a dynamic adapter where parameters are adjusted input instance-level.

Unlike recent adapter-based approaches <cit.>, iConFormer introduces an input-Conditioned Network (iCoN) that dynamically generates the parameters for each input feature in the adapter. This approach enables for effectively capturing task-specific and local details while keeping the number of learnable parameters small. The effectiveness of our method is evidenced by the quantitative analysis in Figure <ref>, which shows that iConFormer captures task-specific information more effectively than conventional methods. We also analyze the capability to capture fine-grained details by visualizing attention maps in Figure <ref>. 

iConFormer achieves performance competitive to full fine-tuning (FFT) on both classification and dense prediction tasks including monocular depth estimation, semantic segmentation, and instance segmentation with only additional 1.6% to 2.8% backbone parameters. Our method even surpasses FFT for the image classification on CIFAR100 <cit.> and the challenging instance segmentation task on COCO <cit.>. Additionally, iConFormer also outperforms conventional PEFT methods for monocular depth estimation task on NYU-v2 <cit.>, demonstrating the effective utilization of pre-trained backbone parameters with additional learnable parameters dynamically finetuned for specific tasks. 
In summary, our contributions are threefold:







  * We propose iConFormer to enhance representation learning by dynamically adjusts only a small subset of parameters conditioned on input instances in the PEFT framework.



  * We demonstrate that iConFormer effectively captures fine-grained details with input-Conditioned Network (iCoN), leading to substantial improvements in dense prediction tasks.


  * Through comprehensive experiments on classification, monocular depth estimation, instance segmentation, and semantic segmentation, we show that iConFormer achieves remarkable performance by tuning only 1.6% to 2.8% of the Transformer backbone parameters.








§ RELATED WORK




  §.§.§ Transformer in Vision.
 Transformers, initially designed for Natural Language Processing (NLP) tasks such as machine translation <cit.> and text generation <cit.>, have achieved significant success in these areas. This success has led to a shift towards computer vision, starting with the Vision Transformer (ViT) <cit.>. Subsequently, various Transformer-based models <cit.> have achieved notable advancements in tasks including image classification <cit.>, semantic segmentation <cit.>, object detection <cit.>, image restoration <cit.>, and depth estimation <cit.>. Furthermore, transformers have significantly advanced vision recognition through large-scale pretraining <cit.>. However, their larger size compared to previously prevalent CNN backbones presents challenges for fine-tuning on specific tasks. In this context, our work explores methods to adapt pre-trained transformers into target tasks in a more effective and efficient way.




  §.§.§ Parameter Efficient Fine Tuning.
 Parameter Efficient Fine-Tuning (PEFT) methods enable the adaptation of large self-supervised pre-trained models <cit.> to specific tasks without the need to train the entire model. In Natural Language Processing (NLP), notable approaches include adapter methods <cit.>, which integrate small modules into the model while keeping the pre-trained parameters frozen, with only the added modules being fine-tuned. Additionally, other methods involve tuning specific components, such as bias or normalization layers <cit.>, utilizing learnable prompt tokens <cit.>, or applying low-rank matrix approximations <cit.> to update parameters more efficiently.

In computer vision, PEFT techniques inspired by NLP have shown significant progress. For instance, VPT <cit.> is the first method to apply prompt tuning approaches to visual classification tasks. AdaptFormer <cit.> employs a parallel adapter framework to enhance the effectiveness of parameter efficient fine-tuning for visual classification. KAdaptation <cit.> optimizes the adapter using Kronecker products. In addition, LoRand <cit.> employs multi-branch low-rank adapters to achieve impressive performance on dense prediction tasks. Our work focuses on adapter methods, proposing new approaches that aim to achieve performance comparable to or exceeding that of full fine-tuning across various downstream tasks.



§ PRELIMINARY



 §.§ Vision Transformer and its Variants


Vision Transformer (ViT) <cit.>, modified from the Transformer <cit.> proposed in NLP, integrates image patches and positional encodings to capture spatial information. It consists of a patch embedding layer and multiple sequential encoder blocks, as depicted in Fig. <ref>(a).
Given a batch of images x ∈ℝ^B× H× W× 3, the patch embedding layer transforms x into sequential patches x_p∈ℝ^B× M × (P^2C), where H× W is an image resolution, and B is a batch size. P× P is the resolution of an image patch, C is the output channel, and M = HW / P^2 is the number of image patches. The patches are linearly embedded into D dimensions to generate the final input x_in∈ℝ^B × M × D.





In the Transformer encoder block, x_in is first normalized using LayerNorm <cit.>, and then processed by a multi-head self-attention layer (MHSA). The output is combined with x_in via a residual connection:

    x́_́íń = Attention(LN(x_in))+x_in .

Next, x́_́íń is normalized and passed through the MLP layer, followed by residual connection:

    x̃ = MLP(LN (x́_́íń)) , 
       x_out = x̃ + x́_́íń .


This process is repeated N times in the encoder block.
In ViT, the self-attention mechanism captures global features by evaluating relationships between all image patches, enabling a comprehensive understanding of complex dependencies. Advancing this approach, Swin Transformer <cit.> introduces hierarchical attention with shifted windows, which enhances both computational efficiency and feature representation. Other variants <cit.> leverage multi-scale feature extraction for specific vision tasks, demonstrating the adaptability of Transformer models to diverse computer vision challenges.




 §.§ PEFT Methods

Parameter efficient fine-tuning (PEFT) methods, such as prompt tuning <cit.>, low-rank adaptation <cit.>, and adapters <cit.>, are designed to reduce the number of trainable parameters needed for fine-tuning large models. 
Here, we briefly review adapter-based PEFT methods related to our approach, which will be detailed in the following section. 
Adapters introduce small, trainable modules between the layers of the pre-trained model. For instance, if the original model's transformation is defined as h = W · x, adapters modify this operation to h = W · x + A · g(B · x), where B represents the down-projection of the input features, A indicates the up-projection back to the original space, and g(·) denotes a non-linearity. These approaches allow for effective task adaptation while minimizing the number of parameters that need to be updated.








§ PROPOSED METHOD


Conventional adapter-based methods <cit.> rely on static parameter tunings, which may constrain their ability to capture diverse input features with high precision. To address these limitations, we propose iConFormer, a novel framework incorporating the input-Conditioned Network (iCoN), as illustrated in Figure <ref> (b). Unlike conventional adapters, iConFormer dynamically generates convolutional kernels tailored to the unique characteristics of each input.
This dynamic generation facilitates more accurate and flexible feature extraction, 
introducing locality inductive biases into the pre-trained Transformer encoder. 
This approach enables the model to effectively capture both global and local features within a parameter-efficient framework. Consequently, iConFormer not only improves the model’s performance but also ensures a more precise adaptation to varying input conditions.



 §.§ Input-Conditioned Network (iCoN)





The input-Conditioned Network (iCoN) is a key component of the iConFormer framework. Inspired by the concept of dynamic filter networks <cit.>, iCoN employs a channel-wise mechanism to generate convolutional kernels that are tailored to the unique characteristics of each input. This approach enhances parameter efficiency while maintaining the flexibility needed to capture diverse features effectively. Formally, the iCoN module generates channel-wise convolutional kernels using input features from the MLP output, denoted as x̃∈ℝ^B × M × D of (<ref>). First, the feature x̃ is down-projected to d channels and reshaped from M into the spatial dimensions Ĥ×Ŵ, which correspond to H/P × W/P. This reshaping effectively rearranges the patches into a spatial grid to produce the feature x̂∈ℝ^B ×Ĥ×Ŵ× d. Subsequently, the iCoN module generates dynamic convolutional kernels from the reshaped feature x̂. This process is mathematically represented as:

    x̂ = Reshape(Down(x̃)) ,


    W_dynamic = iCoN(x̂) ,

where W_dynamic∈ℝ^B × D × K × K is the dynamically generated kernel and K is the kernel size.  For our implementation, we set K to 3 and d to 64 (d ≪ D). The dynamically generated convolution kernel W_dynamic is then applied to feature x̂, enabling a more precise capture of each input’s distinct features.
 Afterward, the feature is reshaped back to ℝ^B × M × d, followed by applying a non-linear activation function and up-projection to restore the channel dimension to the original size D. This process produces the final output of adapter as follows:


    x_A = Up( σ (Reshape(x̂⊗ W_dynamic))) ,

where ⊗ denotes the channel-wise convolution operation and σ represents the GeLU activation function.
To further enhance model robustness, the features obtained from the adapter and attention layers are combined with residual connections:

    x_out = γ· x_A+x́_́íń ,

where γ is a weight that adjusts the impact of the adapter features. This weight is a learnable scalar, optimized during training.

By leveraging dynamically generated convolutional kernels, iConFormer refines the capture of input-specific features compared to conventional adapters, while maintaining parameter efficiency. This efficiency is achieved by employing a consistent set of parameters within the iCoN to generate convolutional kernels, even though the kernels are dynamically adapted for each specific input. Furthermore, the efficiency is enhanced by generating these dynamic kernels on a channel-wise basis, thereby minimizing the number of tunable parameters while preserving effective adaptation. 




Consequently, iConFormer strikes a balance between effective feature extraction and computational efficiency relative to conventional PEFT methods, addressing their limitations and providing a robust solution for parameter-efficient adaptation.



 §.§ Visual Analysis of Local Representation

To evaluate the performance of capturing both local and global information, Fig. <ref> visualizes the attention maps that offer insights on where the model focuses on. The attention maps are generated using attention rollout <cit.> with AdaptFormer <cit.> and iConFormer, employing Swin Transformer <cit.> as the backbone. Attention rollout computes token attentions by recursively multiplying attention matrices across layers, revealing how attention is distributed across different regions of the image. AdaptFormer adopts a standard pipeline of conducting down-projection, non-linear activation, and up-projection using static weight parameters that are not dynamically adjusted conditioned on input features. While the attention maps of AdaptFormer exhibit ambiguous and scattered attention distributions, with limitations in capturing fine-grained local features such as object edges, the attention maps of iConFormer are significantly more focused and well-aligned with objects.





iConFormer utilizes Input-Conditioned Network (iCoN) to dynamically generate convolutional filters tailored to the input features. These convolutional kernels enable iConFormer to capture detailed spatial features while preserving overall contextual awareness. By focusing on these salient details, iConFormer demonstrates significant improvements in processing complex input data, leading to enhanced accuracy and robustness in dense prediction tasks.



























§ EXPERIMENTS




 §.§ Experimental Settings






































  §.§.§ Datasets and Downstream Tasks.

To evaluate the effectiveness of iConFormer, we conduct comprehensive experiments on both image classification and dense prediction tasks, including monocular depth estimation, semantic segmentation, and instance segmentation. Implementation details can be found in the supplementary material. The datasets used in the experiments are as follows:


  * Image Classification: CIFAR-100 dataset <cit.> consists of 50,000 training images and 10,000 validation images, each with a resolution of 32×32, categorized into 100 classes. The SVHN dataset <cit.> includes over 600,000 labeled images for digit classification, comprising 73,257 training samples, 26,032 test samples, and 531,131 additional training images. The Food-101 dataset <cit.> contains 101,000 images across 101 food categories, with each category having 750 training samples and 250 test samples.

  * Monocular Depth Estimation:  NYU-v2 <cit.> with diverse indoor scenes and KITTI <cit.> with high-resolution outdoor driving scenes are benchmark datasets for depth estimation.

For experiments, we used the standard splits and evaluate using Root Mean Square Error (RMSE). NYU-v2 images were cropped to 352 × 352 pixels, while KITTI images were cropped to 480 × 480 pixels.

  * Semantic Segmentation: ADE20K <cit.> is a widely used semantic segmentation dataset with 20,000 training and 2,000 validation images. For our experiments, we utilized UperNet <cit.> as the framework and evaluated performance using the mean Intersection over Union (mIoU) metric.

  * Instance Segmentation: MS COCO <cit.> is a prominent dataset for instance segmentation, with 118,000 training and 5,000 validation images. We used Cascade Mask R-CNN <cit.> as a task-specific decoder and measured performance with Average Precision for bounding boxes (AP_Box) and masks (AP_Mask).






  §.§.§ Pretrained Backbones.

For a fair comparison with FFT baseline and current PEFT methods, we used different pre-trained backbones depending on the tasks. In the semantic segmentation and instance segmentation tasks, Swin Transformer backbones <cit.>, pre-trained on ImageNet-22k dataset <cit.>, were used <cit.>. Specifically, we used the Swin-Large backbone for semantic segmentation and the Swin-Base backbone for instance segmentation. For the monocular depth estimation, we utilized the standard Swin-V2-Base backbone <cit.> pre-trained using the MIM <cit.>. For the classification task, we adopted the ViT backbone <cit.> pre-trained using MAE <cit.>.







  §.§.§ Baseline methods.


For the image classification task, we used the same set of comparison models as in <cit.>, and additionally included the Adapter method from <cit.>. In the monocular depth estimation, we included comparisons with partial tuning such as BiTFiT <cit.>, LN-Tuning <cit.>. We also evaluated against Partial-l <cit.>, which fine-tunes only the final block of the backbone and parameters outside the backbone. For adapter-based methods, we compared with recent approaches, including Adapter <cit.>, AdaptFormer <cit.>, LoRA <cit.>, and LoRand <cit.>. In the semantic segmentation and instance segmentation tasks, we configured the Adapter model under the same settings as described in <cit.> to ensure a fair comparison. Across all tasks, we also included full fine-tuning (FFT) for performance evaluation. 




 §.§ Main Results










  §.§.§ Image classification.

We evaluate various fine-tuning approaches using backbones pre-trained via self-supervised learning paradigms <cit.>, as detailed in Table <ref>. The results demonstrate that iConFormer consistently outperforms linear probing, Visual Prompt Tuning (VPT) methods, and recently proposed adapter-based techniques. Specifically, iConFormer achieves performance improvements of 4.5%, 3.36%, and 4.99% over VPT on the image benchmarks CIFAR-100, SVHN, and Food-101, respectively. Furthermore, when compared to recent adapter-based methods such as Adapter and AdaptFormer, iConFormer shows up to 1.04%, 0.49%, and 1.08% higher accuracy, respectively. Notably, iConFormer also surpasses the full fine-tuning approach by more than 1% Top-1 accuracy on the CIFAR-100 dataset. In summary, iConFormer is not only highly efficient compared to other adapter-based methods but also achieves superior performance with up to 2% of the parameters used in full fine-tuning.





  §.§.§ Monocular Depth Estimation.

Table <ref> present the performance results for the NYU-v2 datasets. As shown in the tables, iConFormer outperforms other parameter efficient fine-tuning methods in all metrics, with the RMSE value being within 0.04 of the full fine-tuning performance. Moreover, iConFormer shows an RMSE improvement of up to 0.2 compared to partial tuning methods, and an enhancement of up to 0.3 RMSE compared to adapter-based methods such as Adapter, AdaptFormer, and LoRA. These results suggest that iConFormer’s capability to generate and apply input-conditioned kernels significantly contributes to its precision in depth estimation tasks. The performance evaluation on the KITTI dataset is
presented in the supplemental material.




  §.§.§ Semantic Segmentation.

We present the results of the semantic segmentation task on the ADE20K dataset in Table <ref>. By fine-tuning fewer than 3.3 million backbone parameters, iConFormer achieves 50.82% mIoU on ADE20K, which is approximately 0.7% lower than Full fine-tuning. Moreover, iConFormer requires fewer parameters for tuning compared to previously proposed adapter-based methods while still achieving superior performance. These results suggest that iConFormer effectively utilizes a limited subset of parameters to capture task-specific information and learn detailed features.




  §.§.§ Instance Segmentation.

Table <ref> presents the instance segmentation results on the COCO dataset. iConFormer demonstrates significant performance gains by training only 2.78% of the total backbone parameters, surpassing both existing adapter-based methods and full fine-tuning. Specifically, it achieves a 0.8% improvement in AP_box and a 0.7% improvement in AP_mask compared to full fine-tuning. These results effectively reveal the advantages of the proposed method and demonstrate its superiority over full fine-tuning in terms of both storage efficiency and performance. Additionally, these findings suggest that iConFormer optimizes resource utilization through its dynamic kernel approach.





 §.§ Ablation Studies


We conducted ablation studies to explore various aspects of iConFormer and identify the key factors that contribute to its performance. All ablation experiments were performed using the dense tasks.



  §.§.§ input-Conditioned Kernel Size.

In Table <ref>, we conducted an ablation study on the KITTI dataset by varying the size of the input-conditioned kernel. We experimented with kernel sizes of 3×3, 5×5, and 7×7. The results demonstrate that iConFormer performs well with a 3×3 kernel across most evaluation metrics. Notably, the RMSE slightly improves as the kernel size decreases, with 3×3 kernel achieving the lowest RMSE. This observation indicates that 3×3 kernel captures essential local features effectively while maintaining computational efficiency. Given that the performance across kernel sizes is quite similar, 3×3 kernel was adopted for its efficiency, providing a balanced trade-off between accuracy and computational cost for the input-conditioned kernels of iCoN.







  §.§.§ iConFormer Configuration.

We investigated the iConFormer configuration by comparing its sequential and parallel instances, as illustrated in Figure <ref>, where the distinction is based on the placement of the Adapter within the Transformer block. As demonstrated in Table <ref>, the sequential form of iConFormer significantly outperforms the parallel form across all dense tasks. The reason might be: 
(1) the sequential design processes each layer's output in a progressive manner, facilitating deeper feature representations and gradual refinement of complex patterns; (2) the parallel design processes outputs simultaneously, which results in limited inter-layer interaction, weakening the information flow and hindering the model’s capacity to capture intricate features. Therefore, we adopted the sequential design as the default configuration for iConFormer, given its demonstrated superior performance.




§ CONCLUSION

In this work, we have presented iConFormer that leverages a parameter-efficient input-conditioned adapter to effectively capture task-specific features and local information with a limited number of learnable parameters in fine-tuning the models for various downstream tasks. iConFormer demonstrates performance comparable to full fine-tuning across image classification, monocular depth estimation, semantic segmentation, and instance segmentation tasks, by tuning only 1.6% to 2.8% of the backbone parameters. iConFormer effectively addresses the limitations of conventional adapter methods and provides superior performances in all tasks. Although our current focus is on vision recognition tasks, we plan to extend iConFormer to other domains such as natural language processing and multi-modal tasks in future work. We anticipate that this extension will inspire further research into efficient adaptation methods and contribute to developing robust solutions across a variety of applications.



