LMLT: Low-to-high Multi-Level Vision Transformer for Image Super-Resolution
Jeongsoo Kim Jongho Nang Junsuk Choe


preparation
    Euclid Collaboration: J. Adamek0000-0002-0723-6740julian.adamek@uzh.ch<ref>
B. Fiorini0000-0002-0092-4321<ref>
M. Baldi0000-0003-4145-1943<ref>,<ref>,<ref>
G. Brando0000-0003-0805-1905<ref>
M.-A. Breton<ref>,<ref>,<ref>
F. Hassani0000-0003-2640-4460<ref>
K. Koyama0000-0001-6727-6915<ref>
A. M. C. Le Brun0000-0002-0936-4594<ref>
G. Rácz0000-0003-3906-5699<ref>
H.-A. Winther0000-0002-6325-2710<ref>
A. Casalino0000-0001-6709-5292<ref>
C. Hernández-Aguayo0000-0001-9921-8832<ref>
B. Li0000-0002-1098-9188<ref>
D. Potter0000-0002-0757-5195<ref>
E. Altamura0000-0001-6973-1897<ref>
C. Carbone0000-0003-0125-3563<ref>
C. Giocoli0000-0002-9590-7961<ref>,<ref>
D. F. Mota0000-0003-3141-142X<ref>
A. Pourtsidou0000-0001-9110-5550<ref>,<ref>
Z. Sakr0000-0002-4823-3757<ref>,<ref>,<ref>
F. Vernizzi0000-0003-3426-2802<ref>
A. Amara<ref>
S. Andreon0000-0002-2041-8784<ref>
N. Auricchio0000-0003-4444-8651<ref>
C. Baccigalupi0000-0002-8211-1630<ref>,<ref>,<ref>,<ref>
S. Bardelli0000-0002-8900-0298<ref>
P. Battaglia0000-0002-7337-5909<ref>
D. Bonino0000-0002-3336-9977<ref>
E. Branchini0000-0002-0808-6908<ref>,<ref>,<ref>
M. Brescia0000-0001-9506-5680<ref>,<ref>,<ref>
J. Brinchmann0000-0003-4359-8797<ref>,<ref>
A. Caillat<ref>
S. Camera0000-0003-3399-3574<ref>,<ref>,<ref>
V. Capobianco0000-0002-3309-7692<ref>
V. F. Cardone<ref>,<ref>
J. Carretero0000-0002-3130-0204<ref>,<ref>
S. Casas0000-0002-4751-5138<ref>
F. J. Castander0000-0001-7316-4573<ref>,<ref>
M. Castellano0000-0001-9875-8263<ref>
G. Castignani0000-0001-6831-0687<ref>
S. Cavuoti0000-0002-3787-4196<ref>,<ref>
A. Cimatti<ref>
C. Colodro-Conde<ref>
G. Congedo0000-0003-2508-0046<ref>
C. J. Conselice0000-0003-1949-7638<ref>
L. Conversi0000-0002-6710-8476<ref>,<ref>
Y. Copin0000-0002-5317-7518<ref>
F. Courbin0000-0003-0758-6510<ref>,<ref>,<ref>
H. M. Courtois0000-0003-0509-1776<ref>
A. Da Silva0000-0002-6385-1609<ref>,<ref>
H. Degaudenzi0000-0002-5887-6799<ref>
G. De Lucia0000-0002-6220-9104<ref>
M. Douspis0000-0003-4203-3954<ref>
F. Dubath0000-0002-6533-2810<ref>
X. Dupac<ref>
S. Dusini0000-0002-1128-0664<ref>
M. Farina0000-0002-3089-7846<ref>
S. Farrens0000-0002-9594-9387<ref>
S. Ferriol<ref>
P. Fosalba0000-0002-1510-5214<ref>,<ref>
M. Frailis0000-0002-7400-2135<ref>
E. Franceschi0000-0002-0585-6591<ref>
M. Fumana0000-0001-6787-5950<ref>
S. Galeotta0000-0002-3748-5115<ref>
B. Gillis0000-0002-4478-1270<ref>
P. Gómez-Alvarez0000-0002-8594-5358<ref>,<ref>
A. Grazian0000-0002-5688-0663<ref>
F. Grupp<ref>,<ref>
L. Guzzo0000-0001-8264-5192<ref>,<ref>
S. V. H. Haugan0000-0001-9648-7260<ref>
W. Holmes<ref>
F. Hormuth<ref>
A. Hornstrup0000-0002-3363-0936<ref>,<ref>
S. Ilić0000-0003-4285-9086<ref>,<ref>
K. Jahnke0000-0003-3804-2137<ref>
M. Jhabvala<ref>
B. Joachimi0000-0001-7494-1303<ref>
E. Keihänen0000-0003-1804-7715<ref>
S. Kermiche0000-0002-0302-5735<ref>
A. Kiessling0000-0002-2590-1273<ref>
M. Kilbinger0000-0001-9513-7138<ref>
B. Kubik0009-0006-5823-4880<ref>
M. Kümmel0000-0003-2791-2117<ref>
M. Kunz0000-0002-3052-7394<ref>
H. Kurki-Suonio0000-0002-4618-3063<ref>,<ref>
S. Ligori0000-0003-4172-4606<ref>
P. B. Lilje0000-0003-4324-7794<ref>
V. Lindholm0000-0003-2317-5471<ref>,<ref>
I. Lloro<ref>
G. Mainetti0000-0003-2384-2377<ref>
E. Maiorano0000-0003-2593-4355<ref>
O. Mansutti0000-0001-5758-4658<ref>
O. Marggraf0000-0001-7242-3852<ref>
K. Markovic0000-0001-6764-073X<ref>
M. Martinelli0000-0002-6943-7732<ref>,<ref>
N. Martinet0000-0003-2786-7790<ref>
F. Marulli0000-0002-8850-0303<ref>,<ref>,<ref>
R. Massey0000-0002-6085-3780<ref>
E. Medinaceli0000-0002-4040-7783<ref>
S. Mei0000-0002-2849-559X<ref>
M. Melchior<ref>
Y. Mellier<ref>,<ref>
M. Meneghetti0000-0003-1225-7084<ref>,<ref>
E. Merlin0000-0001-6870-8900<ref>
G. Meylan<ref>
M. Moresco0000-0002-7616-7136<ref>,<ref>
L. Moscardini0000-0002-3473-6716<ref>,<ref>,<ref>
C. Neissner0000-0001-8524-4968<ref>,<ref>
S.-M. Niemi<ref>
C. Padilla0000-0001-7951-0166<ref>
S. Paltani0000-0002-8108-9179<ref>
F. Pasian0000-0002-4869-3227<ref>
K. Pedersen<ref>
W. J. Percival0000-0002-0644-5727<ref>,<ref>,<ref>
V. Pettorino<ref>
S. Pires0000-0002-0249-2104<ref>
G. Polenta0000-0003-4067-9196<ref>
M. Poncet<ref>
L. A. Popa<ref>
L. Pozzetti0000-0001-7085-0412<ref>
F. Raison0000-0002-7819-6918<ref>
A. Renzi0000-0001-9856-1970<ref>,<ref>
J. Rhodes0000-0002-4485-8549<ref>
G. Riccio<ref>
E. Romelli0000-0003-3069-9222<ref>
M. Roncarelli0000-0001-9587-7822<ref>
R. Saglia0000-0003-0378-7032<ref>,<ref>
A. G. Sánchez0000-0003-1198-831X<ref>
D. Sapone0000-0001-7089-4503<ref>
B. Sartoris0000-0003-1337-5269<ref>,<ref>
M. Schirmer0000-0003-2568-9994<ref>
T. Schrabback0000-0002-6987-7834<ref>
A. Secroun0000-0003-0505-3710<ref>
G. Seidel0000-0003-2907-353X<ref>
S. Serrano0000-0002-0211-2861<ref>,<ref>,<ref>
C. Sirignano0000-0002-0995-7146<ref>,<ref>
G. Sirri0000-0003-2626-2853<ref>
L. Stanco0000-0002-9706-5104<ref>
J. Steinwagner0000-0001-7443-1047<ref>
P. Tallada-Crespí0000-0002-1336-8328<ref>,<ref>
D. Tavagnacco0000-0001-7475-9894<ref>
I. Tereno<ref>,<ref>
R. Toledo-Moreo0000-0002-2997-4859<ref>
F. Torradeflot0000-0003-1160-1517<ref>,<ref>
I. Tutusaus0000-0002-3199-0399<ref>
E. A. Valentijn<ref>
L. Valenziano0000-0002-1170-0104<ref>,<ref>
T. Vassallo0000-0001-6512-6358<ref>,<ref>
G. Verdoes Kleijn0000-0001-5803-2580<ref>
A. Veropalumbo0000-0003-2387-1194<ref>,<ref>,<ref>
Y. Wang0000-0002-4749-2984<ref>
J. Weller0000-0002-8282-2010<ref>,<ref>
G. Zamorani0000-0002-2318-301X<ref>
E. Zucca0000-0002-5845-8132<ref>
A. Biviano0000-0002-0857-0732<ref>,<ref>
C. Burigana0000-0002-3005-5796<ref>,<ref>
M. Calabrese0000-0002-2637-2422<ref>,<ref>
D. Di Ferdinando<ref>
J. A. Escartin Vigo<ref>
G. Fabbian0000-0002-3255-4695<ref>,<ref>
F. Finelli0000-0002-6694-3269<ref>,<ref>
J. Gracia-Carpio<ref>
S. Matthew0000-0001-8448-1697<ref>
N. Mauri0000-0001-8196-1548<ref>,<ref>
A. Pezzotta0000-0003-0726-2268<ref>
M. Pöntinen0000-0001-5442-2530<ref>
V. Scottez<ref>,<ref>
M. Tenti0000-0002-4254-5901<ref>
M. Viel0000-0002-2642-5707<ref>,<ref>,<ref>,<ref>,<ref>
M. Wiesmann0009-0000-8199-5860<ref>
Y. Akrami0000-0002-2407-7956<ref>,<ref>
V. Allevato0000-0001-7232-5152<ref>
S. Anselmi0000-0002-3579-9583<ref>,<ref>,<ref>
M. Archidiacono0000-0003-4952-9012<ref>,<ref>
F. Atrio-Barandela0000-0002-2130-2513<ref>
A. Balaguera-Antolinez0000-0001-5028-3035<ref>,<ref>
M. Ballardini0000-0003-4481-3559<ref>,<ref>,<ref>
A. Blanchard0000-0001-8555-9003<ref>
L. Blot0000-0002-9622-7167<ref>,<ref>
H. Böhringer0000-0001-8241-4204<ref>,<ref>,<ref>
S. Borgani0000-0001-6151-6439<ref>,<ref>,<ref>,<ref>
S. Bruton0000-0002-6503-5218<ref>
R. Cabanac0000-0001-6679-2600<ref>
A. Calabro0000-0003-2536-1614<ref>
B. Camacho Quevedo0000-0002-8789-4232<ref>,<ref>
G. Cañas-Herrera0000-0003-2796-2149<ref>,<ref>
A. Cappi<ref>,<ref>
F. Caro<ref>
C. S. Carvalho<ref>
T. Castro0000-0002-6292-3228<ref>,<ref>,<ref>,<ref>
K. C. Chambers0000-0001-6965-7789<ref>
S. Contarini0000-0002-9843-723X<ref>
A. R. Cooray0000-0002-3892-0190<ref>
G. Desprez0000-0001-8325-1742<ref>
A. Díaz-Sánchez0000-0003-0748-4768<ref>
J. J. Diaz<ref>
S. Di Domizio0000-0003-2863-5895<ref>,<ref>
H. Dole0000-0002-9767-3839<ref>
S. Escoffier0000-0002-2847-7498<ref>
A. G. Ferrari0009-0005-5266-4110<ref>,<ref>
P. G. Ferreira0000-0002-3021-2851<ref>
I. Ferrero0000-0002-1295-1132<ref>
A. Finoguenov0000-0002-4606-5403<ref>
F. Fornari0000-0003-2979-6738<ref>
L. Gabarra0000-0002-8486-8856<ref>
K. Ganga0000-0001-8159-8208<ref>
J. García-Bellido0000-0002-9370-8360<ref>
T. Gasparetto0000-0002-7913-4866<ref>
V. Gautard<ref>
E. Gaztanaga0000-0001-9632-0815<ref>,<ref>,<ref>
F. Giacomini0000-0002-3129-2814<ref>
F. Gianotti0000-0003-4666-119X<ref>
G. Gozaliasl0000-0002-0236-919X<ref>
C. M. Gutierrez0000-0001-7854-783X<ref>
A. Hall0000-0002-3139-8651<ref>
H. Hildebrandt0000-0002-9814-3338<ref>
J. Hjorth0000-0002-4571-2306<ref>
A. Jimenez Muñoz0009-0004-5252-185X<ref>
S. Joudaki0000-0001-8820-673X<ref>
J. J. E. Kajava0000-0002-3010-8333<ref>,<ref>
V. Kansal0000-0002-4008-6078<ref>,<ref>
D. Karagiannis0000-0002-4927-0816<ref>,<ref>
C. C. Kirkpatrick<ref>
S. Kruk0000-0001-8010-8879<ref>
J. Le Graet0000-0001-6523-7971<ref>
L. Legrand0000-0003-0610-5252<ref>
J. Lesgourgues0000-0001-7627-353X<ref>
T. I. Liaudat0000-0002-9104-314X<ref>
A. Loureiro0000-0002-4371-0876<ref>,<ref>
G. Maggio0000-0003-4020-4836<ref>
M. Magliocchetti0000-0001-9158-4838<ref>
F. Mannucci0000-0002-4803-2381<ref>
R. Maoli0000-0002-6065-3025<ref>,<ref>
C. J. A. P. Martins0000-0002-4886-9261<ref>,<ref>
L. Maurin0000-0002-8406-0857<ref>
R. B. Metcalf0000-0003-3167-2574<ref>,<ref>
M. Migliaccio<ref>,<ref>
M. Miluzio<ref>,<ref>
P. Monaco0000-0003-2083-7564<ref>,<ref>,<ref>,<ref>
A. Montoro0000-0003-4730-8590<ref>,<ref>
A. Mora0000-0002-1922-8529<ref>
C. Moretti0000-0003-3314-8936<ref>,<ref>,<ref>,<ref>,<ref>
G. Morgante<ref>
S. Nadathur0000-0001-9070-3102<ref>
L. Patrizii<ref>
V. Popa0000-0002-9118-8330<ref>
P. Reimberg0000-0003-3410-0280<ref>
I. Risso0000-0003-2525-7761<ref>
P.-F. Rocci<ref>
M. Sahlén0000-0003-0973-4804<ref>
E. Sarpa0000-0002-1256-655X<ref>,<ref>,<ref>
A. Schneider0000-0001-7055-8104<ref>
M. Sereno0000-0003-0302-0325<ref>,<ref>
A. Silvestri0000-0001-6904-5061<ref>
A. Spurio Mancini0000-0001-5698-0990<ref>,<ref>
K. Tanidis<ref>
C. Tao0000-0001-7961-8177<ref>
N. Tessore0000-0002-9696-7931<ref>
G. Testera<ref>
R. Teyssier0000-0001-7689-0933<ref>
S. Toft0000-0003-3631-7176<ref>,<ref>
S. Tosi0000-0002-7275-9193<ref>,<ref>
A. Troja0000-0003-0239-4595<ref>,<ref>
M. Tucci<ref>
C. Valieri<ref>
J. Valiviita0000-0001-6225-3693<ref>,<ref>
D. Vergani0000-0003-0898-2216<ref>
G. Verza0000-0002-1886-8348<ref>,<ref>
P. Vielzeuf0000-0003-2035-9339<ref>
N. A. Walton0000-0003-3983-8778<ref>
    Received XXX; accepted ZZZ
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
Recent Vision Transformer (ViT)-based methods for Image Super-Resolution have demonstrated impressive performance. However, they suffer from significant complexity, resulting in high inference times and memory usage. Additionally, ViT models using Window Self-Attention (WSA) face challenges in processing regions outside their windows. To address these issues, we propose the Low-to-high Multi-Level Transformer (LMLT), which employs attention with varying feature sizes for each head. LMLT divides image features along the channel dimension, gradually reduces spatial size for lower heads, and applies self-attention to each head. This approach effectively captures both local and global information. By integrating the results from lower heads into higher heads, LMLT overcomes the window boundary issues in self-attention. Extensive experiments show that our model significantly reduces inference time and GPU memory usage while maintaining or even surpassing the performance of state-of-the-art ViT-based Image Super-Resolution methods. Our codes are availiable at <https://github.com/jwgdmkj/LMLT>.




§ INTRODUCTION


Single Image Super-Resolution (SISR) is a technique that converts low-resolution images into high-resolution ones and has been actively researched in the field of computer vision. Traditional methods, such as nearest neighbor interpolation and bilinear interpolation, were used in the past, but recent super-resolution research has seen significant performance improvements, particularly through CNN-based methods <cit.> and Vision Transformer (ViT)-based methods <cit.>.

Since the introduction of SRCNN <cit.>, CNN-based image super-resolution architectures have advanced by utilizing multiple convolutional layers to understand contexts at various scales. These architectures deliver this understanding through residual and/or dense connections <cit.>. 

However, super-resolution using CNNs faces several issues in terms of performance and efficiency. Firstly, CNN-based models can become excessively complex and deep to improve performance, leading to increased model size and memory usage <cit.>. To mitigate this, several models share parameters between modules <cit.>, but this approach does not guarantee efficiency during inference <cit.>. SAFMN <cit.> addresses the balance between accuracy and complexity by partitioning image features using a multi-head approach <cit.> and implementing non-local feature relationships at various scales. However, it struggles to capture long-range dependencies due to limited kernel sizes.

ViT-based models have shown superior performance compared to CNN-based models by effectively modeling global context interactions <cit.>. For example, SwinIR <cit.> utilized the Swin Transformer <cit.> for image super-resolution, demonstrating the effectiveness of the transformer architecture. Subsequently, hybrid models combining ViT and CNN have been proposed, achieving significant performance increases <cit.>. 

However, ViT models face quadratically increasing computational costs as input size grows <cit.>. To address this, Window Self-Attention (WSA) has been developed, which perform self-attention by dividing the image into windows <cit.>. Despite this, WSA suffers from quality degradation at window boundaries and lacks interaction between windows <cit.>. Additionally, conventional ViT-based models stack self-attention layers in series <cit.>, which significantly increases computational load and inference time.



	

	










In this paper, we propose LMLT (Low-to-high Multi-Level Transformer) to improve efficiency during inference while maintaining performance. Similar to SAFMN, our approach uses a multi-head method <cit.> to split image features and apply pooling to each feature. Each head applies the self-attention mechanism. Unlike conventional self-attention blocks, which stack self-attention layers in series (Figure <ref>(a)), we stack them in parallel to reduce computation (Figure <ref>(b)). This means we integrate the number of heads and layers (depth) into a single mechanism. Note that we call the head with the most pooling the lower head, and the number of pooling applications decreases as we move to the upper heads. 

Since the window size is the same for all heads, the upper heads focus on smaller areas and effectively capture local context. In contrast, the lower heads focus on larger areas and learn more global information. This approach allows us to dynamically capture both local and global information. Additionally, we introduce a residual connection <cit.> to pass global information from the more pooled lower heads to the less pooled upper heads. This enables the windows of the upper heads to view a wider area, thereby resolving the cross-window communication problem.

Trained on DIV2K <cit.> and Flickr2K <cit.>, our extensive experiments demonstrate that ViT-based models can effectively achieve a balance between model complexity and accuracy. Compared to other state-of-the-art results, our approach significantly reduces memory usage and inference time while enhancing performance. Specifically, our base model with 60 channels and large model with 84 channels decrease memory usage to 38% and 54%, respectively, and inference time to 22% and 19% compared to ViT-based super-resolution models like NGswin <cit.> and SwinIR-light<cit.> at scale × 4 scale. Moreover, our models achieve an average performance increase of 0.076db and 0.152db across all benchmark datasets.




§ RELATED WORKS


CNN-Based Image Super-Resolution is one of the most popular deep learning-based methods for enhancing image resolution. Since SRCNN <cit.> introduced a method to restore high-resolution (HR) images using three end-to-end layers, advancements like VDSR <cit.> and DRCN <cit.> have leveraged deeper neural network structures. These methods introduced recursive neural network structures to produce higher quality results. ESPCN <cit.> significantly improved the speed of super-resolution by replacing bicubic-filter upsampling with sub-pixel convolution, a technique adopted in several subsequent works <cit.>. To address the limited receptive field of CNNs, some researchers incorporated attention mechanisms into super-resolution models to capture larger areas. RCAN <cit.> applied channel attention to adaptively readjust the features of each channel, while SAN <cit.> used a second-order attention mechanism to capture more long-distance spatial contextual information. CSFM <cit.> dynamically modulated channel-wise and spatial attention, allowing the model to selectively emphasize various global and local features of the image. We use the same window size, similar to CNN kernels, but vary the spatial size of each feature. This allows our model to dynamically capture both local and global information by obtaining global information from smaller spatial sizes and local information from larger spatial sizes.

ViT-Based Image Super-Resolution has surpassed the performance of CNN-based models by efficiently modeling long-range dependencies and capturing global interactions between contexts <cit.>. After the success of ViT <cit.> in various fields such as classification <cit.>, object detection <cit.>, and semantic segmentation <cit.>, several models have aimed to use it for low-level vision tasks. IPT <cit.> constructed a Transformer-based large-scale pre-trained model for image processing. However, the complexity of ViT grows quadratically with input size. To mitigate this, many approaches have aimed to reduce computational load while capturing both local and global information. For example, SwinIR <cit.> used the Swin-Transformer <cit.> model for image reconstruction. Restormer <cit.> organized self-attention in the channel direction to maintain global information and achieve high performance in image denoising. HAT <cit.> combined self-attention, which captures representative information, with channel attention, which holds global information. To combine local and global information without adding extra complexity, we add features from lower heads, which contain global information, to upper heads, which contain local information. This enables the windows to see beyond their own area and cover a larger region.

Efficient Image Super-Resolution research focuses on making super-resolution models more efficient. The CNN-based model FSRCNN <cit.> improved on SRCNN <cit.> by removing the bicubic interpolation pre-processing and increasing the scale through deconvolution, greatly speeding up computation. CARN <cit.> reused features at various stages through cascading residual blocks connected in a multi-stage manner. IMDN <cit.> progressively refined features passing through the network. However, improving performance often requires stacking many convolution layers, leading to increased computational load and memory usage.

In contrast, the ViT-based model ELAN <cit.> aimed to enhance spatial adaptability by using various window sizes in self-attention. HNCT <cit.> integrated CNN and Transformer structures to extract local features with global dependencies. NGswin <cit.> addressed the cross-window communication problem of the original Swin Transformer <cit.> by applying Ngram <cit.>. Despite these advances, the considerable computational load of overly deep stacked self-attention mechanisms still constrains the efficiency of ViT-based super-resolution models. To address computational load, we connect self-attention layers in parallel, integrating multi-head and depth (number of layers) to lighten the computation. This, along with reducing the spatial size of features, makes our model more efficient.

Additionally, efforts to lighten networks through methods such as knowledge distillation <cit.>, model quantization <cit.>, or pruning <cit.> have been made. Some approaches differentiate between classical and lightweight image super-resolution models by using the same architecture but varying hyperparameters, such as the number of network blocks or feature channels <cit.>. 








§ PROPOSED METHOD


Overall Architecture (Figure <ref>(a)). First, we use a 3 × 3 convolution to extract shallow-level features from the image. Next, we stack multiple LHS Blocks (Low-to-High Self-attention Blocks) to extract deep-level features. In each LHS Block, the features go through Layer Normalization (LN) <cit.>, our proposed LMLT (Low-to-high Multi-Level Transformer), LN again, and the CCM <cit.>. Residual connections are also employed. Finally, we use a 3 × 3 convolution filter and a pixel-shuffle layer <cit.> to reconstruct high-quality images. For more details on CCM <cit.>, refer to Appendix <ref>.

Low-to-high Multi-Level Transformer (Figure <ref>(b)). LMLT operates within the LHS Block. After features pass through the first LN <cit.>, we divide them into H heads using a Multi-Head approach <cit.> and pool each split feature to a specified size. Specifically, the feature for the uppermost head is not pooled, and as we move to lower heads, the pooling becomes stronger, with the height and width halved for each subsequent head. Each feature then undergoes a self-attention mechanism. The output of the self-attention is interpolated to the size of the upper head's feature and added element-wise (called a low-to-high connection). The upper head then undergoes the self-attention process again, continuing up to the topmost head. Finally, the self-attention outputs of all heads are restored to their original size, concatenated, and merged through a 1 × 1 convolution before being multiplied with the original feature.

Attention Layer (Figure <ref>(c)). In each attention layer, the feature is divided into 𝑁 non-overlapping windows. The dot product of the query and key is calculated, followed by the dot product with the value. LePE <cit.> is used as the Positional Encoding and added to the value. The output is upscaled by a factor of 2 and sequentially passed to the upper head until it reaches the topmost head.

How the Proposed Method Works? Our proposed LMLT effectively captures both local and global regions. As seen in Figure <ref>, even if the window size is the same, the viewing area changes with different spatial sizes of the feature. Specifically, when self-attention is applied to smaller features, global information can be obtained. As the spatial size increases, the red window in the larger feature can utilize information beyond its own limits for self-attention calculation because it has already acquired information from other regions in the previous stage. This combination of lower heads capturing global context and upper heads capturing local context secures cross-window communication. Figure <ref> visualizes the type of information each head captures. From Figure <ref>(a) to  <ref>(d), the features extracted from each head when 𝐻 is assumed to be 4 are visualized by averaging them along the channel dimension. The first head (<ref>(a)) captures relatively local patterns, while fourth head (<ref>(d)) captures global patterns. In Figure <ref>(e), these local and global patterns are combined to provide a comprehensive representation. By merging this with the original feature (<ref>(f)), it emphasizes the parts that are important for super-resolution. 

Computational Complexity. We improve the model's efficiency by connecting self-attention layers in parallel and reducing spatial size. In the proposed model, given a feature 𝐹∈ℝ^𝐻×𝑊×𝐷 and a fixed window size of 𝑀×𝑀, the number of windows in our LMLT is reduced by one-fourth as we move to lower heads, by halving the spatial size of the feature map. Additionally, since each head replaces depth, the channel count is also reduced to 𝐷/ℎ𝑒𝑎𝑑. Therefore, the total computation for each head is given by Equation <ref>. Here, 𝑖 means 𝑖-1th head.


    Ω(LMLT) = 4[hw/4^i(D/ℎ𝑒𝑎𝑑)^2] + 2 [ M^2 hw/4^iD/ℎ𝑒𝑎𝑑].


On the other hand, in WSA <cit.>, the self-attention layers are stacked in series, and the spatial size and channel of the feature do not decrease, so the number of windows remains 𝐻𝑊/𝑀^2, and the channel count stays at 𝐷. Therefore, the total computation amount is shown in Equation <ref>.


    Ω(WSA) = 4hw D^2 + 2M^2 hw D.


Therefore, in our proposed model, if the number of heads is greater than 1, both the number of windows and channels decrease compared to WSA <cit.>, resulting in reduced computational load. The more heads there are, the greater the reduction in computational load.





R0.4
 

  
      
    < g r a p h i c s >
    
      
    < g r a p h i c s >
    
      
    < g r a p h i c s >
    
      
    < g r a p h i c s >
 

           (a)    (b)    (c)  


          
      
    < g r a p h i c s >
     
      
    < g r a p h i c s >
     
      
    < g r a p h i c s >
 

          (d)      (e)    (f) 
      


 
 

 
 Features from each head ((a) to (d)), aggregated feature (e), and feature multiplied with the original feature (f).
 






§ EXPERIMENTS


Datasets. Following previous studies <cit.>, we use DIV2K <cit.>, consisting of 800 images, and Flickr2K <cit.>, consisting of 2,650 images, as training datasets. For testing, we use the Set5 <cit.>, Set14 <cit.>, BSD100 <cit.>, Urban100 <cit.>, and Manga109 <cit.> datasets.

Implementation Details. We categorize our model into four types: a Tiny model with 36 channels, a Small model with 36 channels and 12 blocks, a Base model with 60 channels, and a Large model with 84 channels. First, the low-resolution (LR) images used as training inputs are cropped into 64 × 64 patches. Rotation and horizontal flip augmentations are applied to this training data. The number of blocks, heads, and growth ratio are set to 8 (except for the Small model), 4, and 2, respectively. We use the Adam Optimizer <cit.> with β_1 = 0.9 and β_2 = 0.99, running for 500,000 iterations. The initial learning rate is set to 1×10^-3 and is reduced to at least 1×10^-5 using the cosine annealing scheme <cit.>. To accelerate the speed of our experiments, we set  to  and  to  for the 36-channel model. To account for potential variability in the results due to this setting, we conduct three separate experiments with the LMLT-Tiny model and reported the average of these results. All other experiments are conducted only once.





Evaluation Metrics. The quality of the recovered high-resolution images is evaluated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) <cit.>. These metrics are calculated on the Y channel of the YCbCr color space. To test the efficiency of our model, we follow the method of SAFMN <cit.>, measuring GPU memory consumption (#GPU Mem) and inference time (#AVG Time) for scaling a total of 50 images across various models. #GPU Mem, obtained through PyTorch's , represents the maximum memory consumption during inference, and #AVG Time is the average time per image for inferring a total of 50 LR images at ×2, ×3, and ×4 scales. The results for ×2, ×3, and ×4 scaling are based on upscaling random images of sizes 640×360, 427×240, and 320×180, respectively.



 §.§ Comparisons with State-of-the-Art Methods




Image Reconstruction Comparisons. To evaluate the performance of the proposed model, we compare our models with other state-of-the-art efficient and lightweight SR models at different scaling factors. PSNR, SSIM <cit.>, the number of parameters, and FLOPs are used as the main performance evaluation metrics. Note that FLOPs refer to the computational amount required to create an image with a resolution of 1280×720.






We first compare the LMLT-Base with IMDN <cit.>, LatticeNet <cit.>, RFDN-L <cit.>, SRPN-Lite <cit.>, HNCT <cit.>, FMEN <cit.>, and NGswin <cit.>. Table <ref> shows that our LMLT-Base achieves the best or second-best performance on most benchmark datasets. Notably, we observe a significant performance increase on the Manga109 dataset, while our model uses up to 30% fewer parameters compared to the next highest performing model, NGswin <cit.>, while the PSNR increases by 0.27dB, 0.29dB, and 0.29dB, respectively, at all scales.


Next, we compare the LMLT-Large model with other SR models. The comparison group includes ESRT <cit.>, SwinIR-light <cit.>, ELAN <cit.>, and SRFormer-light <cit.>. As shown in Table <ref>, our large model ranks first or second in performance on most datasets. Among the five test datasets, LMLT-Large shows the best performance for all scales on the Manga109 <cit.> dataset compared to others, and also the best performance on the Set14 <cit.> dataset. Specifically, compared to SRFormer-light <cit.>, which showed the highest performance on Urban100 <cit.> among the comparison group, our model shows performance gains of 0.13dB, 0.24dB, and 0.15dB at each scale on the Manga109 <cit.> dataset. In addition to this, we demonstrate that our model has a significant advantage in inference time and GPU memory occupancy at next paragraph. The comparison results of LMLT-Tiny and LMLT-Small with other state-of-the-art models can be found in Appendix <ref>.



Memory and Running Time Comparisons. To test the efficiency of the proposed model, we compare the performance of our LMLT model against other ViT-based state-of-the-art super-resolution models at different scales. We evaluate LMLT-Base against NGswin <cit.> and HNCT <cit.>, and LMLT-Large against SwinIR-light <cit.> and SRFormer-light <cit.>. The results are shown in Table <ref>.






We observe that LMLT-Base and LMLT-Large is quite efficient in terms of inference speed and memory usage compared to other ViT-based SR models. Specifically, compared to NGswin <cit.>, our LMLT-Base maintains similar performance while reducing memory usage by 61%, 62%, and 61% for ×2, ×3, and ×4 scales, respectively, and decreasing inference time by an average of 78%, 76%, and 78%. Similarly, when comparing SwinIR <cit.> and our LMLT-Large, despite maintaining similar performance, memory usage decreases by 44%, 43%, and 46% for each scale, respectively, and inference time decreases by an average of 87%, 80%, and 81%. This demonstrates both the efficiency and effectiveness of the proposed model.


R0.5
Time consumption from LN to LHSB (Ours) and WSA (SwinIR <cit.>). A RTX 3090 GPU is used.
	
        
        0.5!
    	
    		method            × 2             × 3       × 4  

    		LMLT-Tiny(Ours)     35.28𝑚𝑠       23.21𝑚𝑠      18.36𝑚𝑠   

    		LMLT-Base(Ours)     49.44𝑚𝑠      25.51𝑚𝑠       21.18𝑚𝑠   

    		LMLT-Large(Ours)     68.97𝑚𝑠      32.72𝑚𝑠      22.66𝑚𝑠   

            SwinIR-Light <cit.>     1084.57𝑚𝑠      336.00𝑚𝑠       185.23𝑚𝑠  

    		




Table <ref> shows the time consumption for module in the proposed method and SwinIR-light <cit.>, specifically detailing the time from the first Layer Normalization (LN) <cit.> to the LHSB and WSA. Our LMLT significantly reduces the time required for the self-attention mechanism. Especially, LMLT-Large achieves time reductions of 94% at the ×2 scale, 90% at the ×3 scale, and 88% at the ×4 scale compared to SwinIR. Given the similar performance between SwinIR and our method, this represents a significant increase in efficiency. Note that the inference time for a total of 50 random images is measured. The time measurements are conducted using 's record and , and due to hardware access latency during log output, the time of the modules might be longer than the time report in Table <ref>. Tables comparing the memory usage and inference speed of our LMLT with other models can be found in Appendix <ref>.

Qualitative Comparisons. Figure <ref> illustrates the differences on the Manga109 <cit.> dataset between our model and other models. As shown, our LMLT successfully reconstructs areas with continuous stripes better than other models. Additionally, we include more comparison images, and further compare our proposed models LMLT-Tiny with CARN <cit.>, EDSR <cit.>, PAN <cit.>, ShuffleMixer <cit.> and SAFMN <cit.> on the Urban100 <cit.> dataset at ×4 scale. Detailed results can be seen in Appendix <ref>.







 §.§ Ablation Study


Effects of Low-to-high Connection. We examine the effects of the low-to-high element-wise sum (low-to-high connection) and downsizing elements of our proposed model. As shown in Table <ref>, the low-to-high connection yields significant results. Specifically, on the Urban100 <cit.> dataset, PSNR increases by 0.04 dB to 0.05 dB across all scales, and SSIM <cit.> increases by nearly 0.0011 at the ×4 scale, demonstrating the benefits of including the low-to-high connection. Appendix <ref> visualizes the differences in features on Urban100 <cit.> with and without the low-to-high connection, showing that it significantly reduces the boundary lines between windows. Additionally, experiments adding the proposed low-to-high connection to SAFMN <cit.> for ×2, ×3, and ×4 scales are also provided in Appendix <ref>.









Effects of Multi-Scale Heads. Table <ref> validates the effectiveness of using multiple scales by experimenting with cases where pooling is not applied to any head. Specifically, we compare our proposed LMLT with cases where pooling and the low-to-high connection are not applied, as well as cases where merging is also not performed. The results show that performance is lower in all cases compared to the proposed model. Appendix <ref> demonstrates that when pooling is not applied, the lack of information connection between windows hinders the proper capture of informative features, even though spatial information is retained.

Importance of LHSB, CCM, and MLP. We analyze the impact of LHSB and CCM <cit.> and their interplay. Following the approach in SAFMN <cit.>, we examine performance by individually removing LHSB and CCM <cit.>. Results are shown in the `Module' row of Table <ref>. Removing LHSB reduces the number of parameters by nearly 10%, decreases memory usage to nearly 74%, and drops PSNR by 0.59 dB on the Urban100 <cit.> dataset. Conversely, removing CCM reduces the number of parameters by nearly 90% and PSNR by 1.83 dB. Adding an MLP after the self-attention module, as done in traditional Transformers <cit.>, reduces parameters by about 69% and PSNR by approximately 0.85 dB. To maintain the same number of layers as in the previous two experiments (8 layers), we conducted another experiment with only 4 blocks, resulting in a 49% reduction in parameters and only a 0.55 dB drop in PSNR, indicating the least performance loss. This suggests that the combination of LHSB and CCM effectively extracts features. Additionally, incorporating FMBConv <cit.> reduces parameters by nearly 58% and PSNR by 0.28 dB, while memory usage remains similar.

Importance of Aggregation and Activation. We analyze the impact of aggregating features from each head using a 1×1 convolution or applying activation before multiplying with the original input. Results are shown in the `Act / Aggr' row of Table <ref>. Without aggregation, PSNR decreases by 0.10 dB on the Urban100 <cit.> dataset. If features are directly output without applying activation and without multiplying with the original input, PSNR decreases by 0.12 dB. Omitting both steps leads to an even greater decrease of 0.22 dB, indicating that including both aggregation and activation is more efficient. Conversely, multiplying features directly to the original feature without the activation function improves performance by 0.1 dB. Detailed experimental results are discussed in Appendix <ref>.

Importance of Positional Encoding. Lastly, we examine the role of Positional Encoding (PE) in performance improvement. Results are shown in the `PE' row of Table <ref>. Removing PE results in decreased performance across all benchmark datasets, notably with a PSNR drop of 0.06 dB and an SSIM decrease of 0.0006 on the Urban100 <cit.> dataset. Using RPE <cit.> results in a maximum PSNR increase of 0.03 dB on the Set14 <cit.> dataset, but has little effect on other datasets. Additionally, parameters and GPU memory increase by 5K and 45M, respectively.







§ CONCLUSION


In this paper, we introduced the Low-to-high Multi-Level Transformer (LMLT) for efficient image super-resolution. By combining multi-head and depth reduction, our model addresses the excessive computational load and memory usage of traditional ViT models. In addition to this, LMLT applies self-attention to features at various scales, aggregating lower head outputs to inform higher heads, thus solving the cross-window communication issue. Our extensive experiments demonstrate that LMLT achieves a favorable balance between model complexity and performance, significantly reducing memory usage and inference time while maintaining or improving image reconstruction quality. This makes the proposed LMLT a highly efficient solution for image super resolution tasks, suitable for deployment on resource-constrained devices.



plainnat
    






















§ IMPACT OF NUMBER OF BLOCKS, CHANNELS, HEADS AND DEPTHS


In this section, we analyze how the performance of our proposed model changes based on the number of blocks, heads, channels and depths. 





Impact of Number of Blocks. 
First, We evaluate the performance by varying the number of blocks to 4, 6, 8, 10, and 12. Experiments are conducted on ×2 scale and the performance is evaluated using benchmark datasets, and analyzed in terms of the number of parameters, FLOPs, GPU memory usage, and average inference time. 

As shown in Table <ref>, the increase in the number of parameters, FLOPs and inference time tends to be proportional to the number of blocks, and performance also gradually improves. For the Manga109 <cit.> dataset, as the number of blocks increases from 4 to 12 in increments of 2, PSNR increases by 0.27 db, 0.16 db, 0.10 db, and 0.10 db, respectively. Interestingly, despite the increase in the number of blocks from 4 to 12, the GPU memory usage remains almost unchanged. While the number of parameters nearly triples, the GPU memory usage remains stable, 323.5M to 324.5M. We observe the overall increase in PSNR with the increase in the number of blocks and designate the model with 8 blocks as LMLT-Tiny and the model with 12 blocks as LMLT-Small.





Impact of Number of Channels.
Next, we evaluate how performance changes with the number of channels. Similar to the performance evaluation based on the number of blocks, this experiment evaluates performance using benchmark datasets, the number of parameters, FLOPs, GPU memory usage, and average inference time as performance metrics. 

As shown in Table <ref>, LMLT's performance increases with channels, along with parameters and FLOPs. However, unlike the variations in the number of blocks, increasing the number of channels results in a more significant increase in the number of parameters, FLOPs, and memory usage. Inference time, however, increases proportionally with the number of channels. For instance, with 36 channels, the average inference time is 57.16𝑚𝑠, and when doubled, it requires approximately 108.74𝑚𝑠, nearly twice the time. As the number of channels increases from 24 to 84 in increments of 12, the PSNR on the Urban100 <cit.> dataset increases by 0.42 db, 0.29 db, 0.19 db, 0.13 db, and 0.10 db, respectively. Based on the overall performance increase, we designate the model with 60 channels as the Base model and the model with 84 channels as the Large model. In this context, the Small model has an inference time about 3ms longer than the Base model, but it has fewer parameters, lower memory usage, and fewer FLOPs, thus justifying its designation.


Impact of Number of Heads.




In this paragraph, we compare the performance differences based on the number of heads. In our model, as the number of heads decreases, the channel and the number of downsizing operations for each head decrease. For example, in our baseline with 4 heads and 36 channels, the lowest head has a total of 9 channels and is pooled 3 times. However, if there are 2 heads, the lowest head has 18 channels and is pooled once. Additionally, the maximum pooling times and the number of heads are related to the number of windows and the amount of self-attention computation. According to equation <ref>, as the number of heads decreases, the computation increases. As a result, as the number of heads decreases, the number of parameters, FLOPs, and GPU memory usage increase.

As shown in Table <ref>, the performance with 4 heads and 3 heads is similar across all scales and test datasets. However, when the number of heads is reduced to 1, the performance drops significantly. This difference is particularly noticeable in the Urban100 <cit.> dataset, where at scale ×2, the performance with 4 heads is 32.04 db, whereas with 1 head, it drops to 31.93 db, a decrease of 0.11 db. Additionally, when the scale is ×3 and ×4, the PSNR decreases by 0.05 dB and 0.04 dB, respectively. This indicates that even if the spatial size of all features is maintained with a single head, even though the number of parameters, FLOPs, and channels per head increase, the inability to capture information from other windows can lead to a decline in performance. 


Impact of Number of Depths.




Additionally, we examine how the performance changes when we add more attention modules to our model. The proposed LMLT connects self-attention layers in parallel, where self-attention is calculated in lower heads and then connected to the upper layers. However, a different approach, like other models <cit.>, could be to calculate self-attention multiple times(i.e., in series) before sending it to the upper heads, thus mixing serial and parallel connections.

However, our experimental results indicate that calculating self-attention multiple times in a single head before sending it to the upper heads is not an effective choice. As shown in Table <ref>, increasing the self-attention calculations from once to three times increases the inference time by 87.7% on the Urban100 <cit.> dataset at × 4 scale, but the PSNR only improves by 0.02 dB. Similar trends are observed in other datasets and scales, showing minimal differences in PSNR and SSIM performance. This demonstrates that having one head composed of serial self-attention layers and connecting heads in parallel does not yield good performance relative to inference time.





§ EFFECTS OF LOW-TO-HIGH CONNECTION AND POOLING





Difference between with and without Low-to-high connection




In the Table <ref>, we confirm performance differences when the low-to-high connection is not applied to LMLT. Inspired by this, we also apply low-to-high connections between heads in SAFMN <cit.> and verify the experimental results. Table <ref> shows that adding low-to-high connection to the upper head in SAFMN <cit.> does not yield significant performance differences. Moreover, at the ×4 scale, the SSIM for the Urban100 <cit.> and Set5 <cit.> datasets decreases by 0.0010 and 0.0011, respectively, indicating a reduction in performance.

We then visualize the features of LMLT-Tiny to understand the effect of the low-to-high connection. Each column of Figure <ref> illustrates the original image, the aggregated feature visualization of LMLT-Tiny combining all heads <ref>(a), and the aggregated feature visualization of LMLT-Tiny without low-to-high connection <ref>(b). In  <ref>(b), the images show pronounced boundaries in areas such as stairs, buildings, and the sky. In contrast,  <ref>(a) shows these boundaries as less pronounced. This demonstrates that the low-to-high connection can address the border communication issues inherent in WSA.

Difference between with and without Pooling




We analyze the impact of pooling on the performance of super-resolution. As observed in Table <ref>, even though pooling preserves spatial information, the overall performance decreases when it is not applied. We investigate the reason behind this through feature visualization. Figure <ref> visualizes the features when no pooling is applied to any head in LMLT. The leftmost image is the original Urban100 <cit.> image. Figure <ref>(a) shows the aggregated features of all heads in LMLT-Tiny. Column Figure <ref>(b) visualizes the features without pooling, and Figure <ref>(c) visualizes the features without both pooling and merging, all at the ×4 scale. In  <ref>(b) and  <ref>(c), grid patterns are evident across the images, indicating that the disadvantages of being limited to local windows outweigh the benefits of maintaining the original spatial size.







§ IMPACT OF ACTIVATION FUNCTION






In Table <ref>, we discuss that not applying the activation function GeLU <cit.> might improve performance. Therefore, we experiment with LMLT and LMLT without GeLU <cit.> across various scales and channels to confirm the results.

Table <ref> shows the results for our LMLT and the model without the activation function across different scales and channels. As shown, with 36 channels, there is minimal performance difference across all scales, with the largest being a 0.04 higher PSNR on the Set5 <cit.> ×4 scale when GeLU <cit.> is removed. However, when expanded to 60 channels, our LMLT performs better on most benchmark datasets for both ×3 and ×4 scales. Specifically, on the ×4 scale of the Urban100 <cit.> dataset, PSNR and SSIM are higher by 0.05 dB and 0.0013, respectively. This demonstrates that adding GeLU <cit.> after aggregating features is more beneficial for performance improvement.





§ LAM AND ERF COMPARISONS









To verify whether the proposed LMLT exhibits a wider receptive field, we utilize local attribution map (LAM) <cit.> and effective receptive field (ERF) <cit.>. Specifically, we use LAM to show that our proposed LMLT-Large has a wider receptive field compared to SwinIR-Light <cit.> and SRFormer-Light <cit.>. Detailed visualizations are presented in Figure <ref>. Additionally, SwinIR-NG <cit.> is included for comparison, and we visualize the ERF. The detailed results are shown in Figure <ref>. Through these two analyses, we demonstrate that our proposed model exhibits a wider receptive field than existing ViT-based SR models.




Additionally, we compare the LAM of our proposed model, LMLT-Tiny (Figure <ref>(c)), with a version of the model that does not include pooling for each head (Figure <ref>(a)), and a version where the low-to-high connection is removed (Figure <ref>(b)), demonstrating that our proposed model effectively references a broader region. The results show that, even when the spatial size of the model is maintained without pooling, it fails to process information from a wider area. Moreover, the low-to-high connection proves to be effective in enabling the model to capture information from a larger region.






§ CCM : CONVOLUTIONAL CHANNEL MIXER

R0.5
	
    < g r a p h i c s >

	
	CCM(Convolutional Channel Mixer) proposed in SAFMN <cit.>.  
	



CCM instead of MLP. Since the feed-forward network (FFN) in the original transformer <cit.> is a fully connected layer, we assume that using it in ViT might disrupt the spatial information of the features. Therefore, we apply the convolutional channel mixer (CCM) <cit.> instead, an FFN based on FMBConv <cit.>, to preserve spatial information. CCM is a module that mixes each convolution channel. Specifically, the features pass through two convolution layers. The first layer has a 3 × 3 kernel and expands the channels. Then, GELU <cit.> is applied for non-linear mapping. Finally, a convolution layer with a 1 × 1 kernel restores the channels to their original state. In our method, the features pass through Layer Normalization <cit.>, LMLT, and another Layer Normalization before being input to CCM <cit.>. Detailed structure can be seen in Figure <ref>.




§ COMPARISONS ON LMLT WITH OTHER METHODS








Image Reconstruction comparisons

Here, We first compare the LMLT-Tiny and LMLT-Small with CARN-m, CARN <cit.>, EDSR-baseline <cit.>, PAN <cit.>, LAPAR-A <cit.>, ECBSR-M16C64 <cit.>, SMSR <cit.>, Shuffle-Mixer <cit.>, and SAFMN <cit.>. Table <ref> shows that our LMLT significantly reduces number of parameters and computation overheads while achieving substantial performance gains on various datasets. LMLT-Small performs well on most datasets, and the LMLT-Tiny also performs second and third best on the BSD100 <cit.> and Manga109 <cit.> datasets, except for the Manga109 ×4 SSIM <cit.>. In particular, the number of parameters and FLOPs are the second smallest after SAFMN <cit.>.  













Memory and Running time Comparisons



In this paragraph, we present the memory usage and average inference time of our proposed LMLT compared to other super-resolution methods. Similar to the experimental setup in Table <ref>, #GPU Mem represents the maximum memory usage during inference, measured using PyTorch's . #AVG Time indicates the average time taken to upscale a total of 50 random images by ×2, ×3, and ×4 scales. The experiments were conducted three times, and the average inference time is reported. Each random image has sizes of 640×360 for ×2 scale, 427×240 for ×3 scale, and 320×180 for ×4 scale.

As shown in Table <ref>, our proposed LMLT-Tiny uses less memory at all scales compared to all models except SAFMN <cit.>. Although LMLT-Small requires more inference time compared to other models, its GPU usage is almost similar to LMLT-Tiny, and its performance is significantly superior as demonstrated in Table <ref>.


Qualitative Comparisons







In this paragraph, we examine the qualitative comparisons of the LMLT-Tiny model and other models on the Urban100 <cit.> ×4 scale. The comparison includes CARN <cit.>, EDSR <cit.>, PAN <cit.>, ShuffleMixer <cit.>, and SAFMN <cit.>. The results can be seen in Figure <ref>. As mentioned in section <ref>, we observe that our model reconstructs images with continuous stripes better than other models.

Additionally, we compare our proposed models LMLT-Base and LMLT-Large with IMDN <cit.>, NGswin <cit.>, SwinIR-light <cit.>, and SwinIR-NG <cit.> on the Manga109 <cit.> dataset at×4 scale. As explained earlier in section <ref>, our model shows strength in areas with continuous lines compared to other models. Figure <ref> illustrates the differences between our LMLT-Base, LMLT-Large and other state-of-the-arts models.

