Deformation-Invariant Neural Network and Its Applications in Distorted
  Image Restoration and Analysis
Images degraded by geometric distortions pose a significant challenge to
imaging and computer vision tasks such as object recognition. Deep
learning-based imaging models usually fail to give accurate performance for
geometrically distorted images. In this paper, we propose the
deformation-invariant neural network (DINN), a framework to address the problem
of imaging tasks for geometrically distorted images. The DINN outputs
consistent latent features for images that are geometrically distorted but
represent the same underlying object or scene. The idea of DINN is to
incorporate a simple component, called the quasiconformal transformer network
(QCTN), into other existing deep networks for imaging tasks. The QCTN is a deep
neural network that outputs a quasiconformal map, which can be used to
transform a geometrically distorted image into an improved version that is
closer to the distribution of natural or good images. It first outputs a
Beltrami coefficient, which measures the quasiconformality of the output
deformation map. By controlling the Beltrami coefficient, the local geometric
distortion under the quasiconformal mapping can be controlled. The QCTN is
lightweight and simple, which can be readily integrated into other existing
deep neural networks to enhance their performance. Leveraging our framework, we
have developed an image classification network that achieves accurate
classification of distorted images. Our proposed framework has been applied to
restore geometrically distorted images by atmospheric turbulence and water
turbulence. DINN outperforms existing GAN-based restoration methods under these
scenarios, demonstrating the effectiveness of the proposed framework.
Additionally, we apply our proposed framework to the 1-1 verification of human
face images under atmospheric turbulence and achieve satisfactory performance,
further demonstrating the efficacy of our approach.
Han Zhang, Qiguang Chen, Lok Ming Lui
Deformation-Invariant Neural Network and Its Applications in Distorted Image Restoration and Analysis

###### Abstract

Images degraded by geometric distortions pose a significant challenge to imaging and computer vision tasks such as object recognition. Deep learning-based imaging models usually fail to give accurate performance for geometrically distorted images. In this paper, we propose the deformation-invariant neural network (DINN), a framework to address the problem of imaging tasks for geometrically distorted images. The DINN outputs consistent latent features for images that are geometrically distorted but represent the same underlying object or scene. The idea of DINN is to incorporate a simple component, called the quasiconformal transformer network (QCTN), into other existing deep networks for imaging tasks. The QCTN is a deep neural network that outputs a quasiconformal map, which can be used to transform a geometrically distorted image into an improved version that is closer to the distribution of natural or good images. It first outputs a Beltrami coefficient, which measures the quasiconformality of the output deformation map. By controlling the Beltrami coefficient, the local geometric distortion under the quasiconformal mapping can be controlled. The QCTN is lightweight and simple, which can be readily integrated into other existing deep neural networks to enhance their performance. Leveraging our framework, we have developed an image classification network that achieves accurate classification of distorted images. Our proposed framework has been applied to restore geometrically distorted images by atmospheric turbulence and water turbulence. DINN outperforms existing GAN-based restoration methods under these scenarios, demonstrating the effectiveness of the proposed framework. Additionally, we apply our proposed framework to the 1-1 verification of human face images under atmospheric turbulence and achieve satisfactory performance, further demonstrating the efficacy of our approach.

 Image Restoration, Turbulence Removal, Bijective Transformation, Generative Adversarial Network, Quasiconformal Geometry

## I Introduction

Deep learning methods have made significant strides in the field of imaging and computer vision, allowing us to achieve remarkable results in tasks like image restoration, object recognition, and classification. However, when it comes to degraded images, deep learning methods can face significant challenges. One such category of degraded images is those that are corrupted by geometric distortion, such as atmospheric turbulence or water turbulence. The use of deep learning methods may fail to produce accurate results for such images. For example, in the facial recognition task for images obtained by long-range cameras, the facial structure in the images is often geometrically distorted due to atmospheric turbulence, causing classical classification networks to provide incorrect results [1]. One intuitive approach for solving this problem is to add distorted images to the downstream classification networks for fine-tuning. However, this approach can be expensive due to the typically large size of the downstream network. Additionally, the introduction of extra variance in the data distribution caused by the distorted images may potentially degrade the performance of the tuned neural network. This challenge motivates the development of a framework that can effectively deal with geometrically distorted images, enabling deep learning methods to achieve accurate and reliable results even in challenging conditions.

There are two possible approaches to address this problem. One approach is to integrate a physical model that describes the geometric distortion. However, finding an appropriate physical model to describe the different types of geometric deformations can be challenging. Another approach is to train a deep neural network to describe and correct the geometric distortion. However, training a deep neural network that can handle a wide range of deformations while maintaining control over the geometric properties of the deformation is a challenging task. This difficulty often leads to inaccurate estimations of the required geometric distortion necessary to correct an image, making it crucial to develop a network that can effectively learn spatial deformation with controlled local geometric distortions.

To address the problem of imaging tasks for geometrically distorted images, we propose the deformation-invariant neural network (DINN), a framework that integrates the quasiconformal transformer network (QCTN) into existing deep neural networks. The QCTN is a deep neural network that outputs a quasiconformal map, which can transform a geometrically distorted image into an improved version that is closer to the distribution of natural or good images. The QCTN achieves this by first outputting a Beltrami coefficient, which measures the quasiconformality of the associated deformation map. By
controlling the Beltrami coefficient, the local geometric distortion under the quasiconformal mapping can be controlled. A key feature of the QCTN is its ability to generate a bijective deformation map. The bijectivity holds great importance as it ensures the preservation of the essential characteristics of the original image. Figure 1 provides an illustration of the significance of bijectivity. In Figure 1(a), an image depicting a degraded digit 9 is presented. Our objective is to transform this degraded image into a non-distorted version. However, if a non-bijective deformation is employed, the digit 9 undergoes a topological change and is transformed into the digit 8, as depicted in Figure 1(b). Conversely, when a bijective deformation is utilized, the digit 9 is transformed into a non-distorted digit 9, as shown in Figure 1(c). This example illustrates the crucial role of bijectivity in preserving the fundamental features of the original image.

Utilizing our framework, we have devised an image classification network that excels in accurately classifying distorted images. Our proposed framework has also been applied to the restoration of geometrically distorted images, including images distorted by atmospheric turbulence and water turbulence. Our proposed framework has outperformed existing GAN-based restoration methods, demonstrating its effectiveness. Additionally, we have applied our proposed framework to the 1-1 verification of human face images under strong air turbulence and achieved good performance, further demonstrating the efficacy of our approach. The proposed DINN framework is believed to be an effective model to enhance the performance of deep neural networks in imaging tasks and enable robust and accurate image analysis in various applications.

In summary, the main contributions of this paper are listed below.

* We introduce the deformation-invariant neural network (DINN) framework, designed to handle imaging tasks involving geometrically distorted images. DINN ensures consistent latent features for images capturing the same underlying object or scene. In the DINN framework, we propose the portable Quasiconformal Transformation Network (QCTN) component, enabling the correction of geometric distortions. This allows large pretrained networks to process heavily distorted images without the need for additional tuning, which can be computationally expensive.
* Based on quasiconformal theories, the QCTN component in DINN generates a bijective deformation map, preserving the salient features of the original image. This property leads to more accurate imaging results, ensuring that the restored images maintain their essential characteristics.
* We utilize the DINN framework to design deep neural networks for tackling three imaging tasks. Firstly, leveraging the DINN framework, we develop an image classification network that demonstrates proficiency in accurately classifying distorted images. Secondly, we employ the DINN framework to address the challenging problem of image restoration in the presence of atmospheric or underwater turbulence. The capability of DINN to effectively handle the complex distortions caused by turbulence in the air and water proves to be highly advantageous in this context. Thirdly, we design a deep neural network using DINN framework for 1-1 facial verification tasks involving facial images that have been corrupted by atmospheric turbulence. Through this application, DINN demonstrates its remarkable efficacy in enhancing the accuracy of facial recognition even under adverse conditions.

## II Related Work

In this section, we present a comprehensive overview of the relevant existing literature that closely relates to our work.

### _Computational quasiconformal geometry_

In this work, computational quasiconformal geometry will be applied. Computational quasiconformal geometry has found extensive application in diverse imaging tasks, yielding successful results. Computational quasiconformal geometry has provided a mathematical tool to study and control the geometric distortions under a mapping. In particular, conformal mappings belong to the class of quasiconformal mappings. Conformal mappings have garnered widespread usage in geometry processing, finding application in a multitude of tasks, including but not limited to texture mapping and surface parameterizations [2, 3, 4]. To quantitatively assess local geometric distortions within a mapping, the associated Beltrami coefficient is commonly employed. By manipulating the Beltrami coefficients, effective control over the geometric properties of the mapping can be achieved.

Fig. 1: The significance of bijectivity. (a) A degraded image of the digit 9. (b) The degraded image undergoes a non-bijective deformation resulting in a topological change, transforming the digit 9 into the digit 8. (c) The degraded image undergoes a bijective deformation. The distorted digit 9 is transformed into a non-distorted digit 9.
Consequently, various surface parameterization methods minimizing conformality distortion have been proposed, leveraging the Beltrami coefficient [5, 6]. Besides, quasiconformal mappings have found applications in computational fabrication [7, 8, 9]. Moreover, numerous quasiconformal imaging models have been introduced in recent years to address diverse imaging tasks, such as image registration [10, 11], surface matching [12] and shape prior image segmentation [13, 14].

### _Deformable Convolution_

Deformable convolution has emerged as a promising solution to overcome the limitations of traditional convolution operations in Convolutional Neural Networks (CNNs). One notable approach is the Active Convolution (AC) proposed by Y. Jeon et al. [15]. AC integrates a trainable attention mechanism into the convolution operation, enabling adaptive feature selection for different input instances. Another related technique is the Spatial Transformer Network (STN) [16], which introduces a learnable transformation module capable of warping the input feature map based on learnable parameters. By incorporating an explicit spatial transformation module, the STN allows the network to learn spatial transformations that align the input with the task at hand, resulting in improved performance for tasks like digit recognition and image classification. Building upon the concept of the STN, D. Dai et al. [17] proposed the Deformable Convolution Network (DCN), which extends the idea of spatial transformation to the convolution operation itself. DCN introduces learnable offsets for each position in the convolutional kernel, enabling dynamic adjustment of the sampling locations for each input instance. This leads to enhanced performance in tasks such as object detection and semantic segmentation. However, the original DCN has limitations in handling large deformations and maintaining invariance to occlusion. To address these limitations, researchers have introduced variations such as the Deformable Convolution v2 (DCNv2) [18], which incorporates additional deformable offsets for intermediate feature maps, and the Deformable RoI Pooling (DRoIPool) [17], which extends the DCN to region-based object detection tasks. Furthermore, W. Luo et al. [19] discovered that the contribution of each pixel is not equal to the final results in DCN, highlighting the need for further improvements in the deformable convolution operation to address its limitations and maximize its performance.

### _Image Restoration_

This work incorporates the use of Generative Adversarial Networks (GANs) to address image restoration tasks. GAN models have shown success in image restoration by training a generator to restore degraded images to their original, high-quality versions [20, 21, 22, 23]. The generator learns to produce visually pleasing and realistic images by taking a degraded image as input and generating an output that closely resembles the original image. In the specific area of image deturbulence, which focuses on restoring images distorted by atmospheric or water turbulence, Lau et al. [24] propose a method that utilizes robust principal component analysis (RPCA) and quasiconformal maps. Their approach aims to restore atmospheric turbulence-distorted images. Another relevant work by Thepa et al. [25] presents a deep neural network-based approach for reconstructing dynamic fluid surfaces from various types of images, including monocular and stereo images. Lau et al. [1] propose a novel GAN-based approach specifically designed for restoring and recognizing facial images distorted by atmospheric turbulence. Their method can restore high-quality facial images from distorted inputs and recognize faces under challenging conditions. Li et al. [26] employ a GAN-based model to remove distortions caused by refractive interfaces, such as water surfaces, using only a single image as input. Furthermore, Rai et al. [27] introduce a channel attention mechanism, adapted from [28], into the generator network of their proposed GAN-based model. This mechanism helps the network focus on more relevant features during the restoration process.

## III Mathematical Formulation

In this section, we present the mathematical formulation of our proposed framework, which leverages the principles of quasiconformal geometry. We also provide an overview of the fundamental mathematical concepts related to quasiconformal theories.

### _Problem formulation_

In this subsection, we provide the mathematical formulation of our problem. Given a distorted image \(\tilde{I}:\Omega\rightarrow\mathbb{R}\) or \(\mathbb{R}^{3}\), \(\tilde{I}\) can be expressed as:

\[\tilde{I}=I\circ\tilde{f}+\epsilon, \tag{1}\]

where \(I\) is the original clean image without any refractive distortion, and \(\epsilon\) represents the additive noise. Here, \(\tilde{f}\) represents the geometric distortion imposed on \(I\).

For most imaging tasks, the input image is assumed to be clean or undistorted. The clean image is fed into a suitable algorithm or deep neural network, which produces the desired imaging results. For example, in a classification task, we use a classifier network \(\mathcal{N}\) that takes an image as input and outputs its predicted category. Usually, \(\mathcal{N}\) is trained on a dataset of
normal, undistorted images with a distribution denoted as \(P_{\text{clear}}\). When the distorted image \(\tilde{I}\) is used as input to \(\mathcal{N}\), \(\mathcal{N}(\tilde{I})\) is likely to produce an incorrect prediction since \(\tilde{I}\) significantly deviates from \(P_{\text{clear}}\). One possible solution is to adjust the parameters in the deep neural network using a training dataset of distorted images. However, in most cases, the network \(\mathcal{N}\) is so large that it becomes expensive to include these distorted images for fine-tuning. Additionally, the introduction of extra variance in the data distribution caused by the distorted images may potentially degrade the performance of the tuned neural network.

To address this problem, our strategy is to learn a deformation map \(f:\Omega\rightarrow\Omega\) to deform \(\tilde{I}\) such that the deformed image \(I^{\prime}=\tilde{I}\circ f\) is closer to the distribution \(P_{\text{clear}}\). In other words, \(I^{\prime}\sim P_{\text{clear}}\). Essentially, \(f\approx\tilde{f}^{-1}\). To solve this problem, we propose the Quasiconformal Transformer Network (QCTN) to learn the deformation \(f\). The QCTN is lightweight, making it more cost-effective than retraining \(\mathcal{N}\) with a large dataset of distorted images.

### _Quasi Conformal Theory_

As discussed in the previous subsection, our strategy for handling distorted images is to train the Quasiconformal Transformer Network (QCTN). Specifically, the QCTN learns a homeomorphic mapping \(f:\Omega\rightarrow\Omega\) to remove the geometric distortion from a distorted image \(\tilde{I}\). Each homeomorphic deformation \(f:\Omega\rightarrow\Omega\) is associated with a geometric quantity known as the _Beltrami coefficient_, defined as:

\[\mu=\frac{\partial f}{\partial z}/\frac{\partial f}{\partial z}, \tag{2}\]

where \(\frac{\partial}{\partial z}=\frac{1}{2}\left(\frac{\partial}{\partial x}-i \frac{\partial}{\partial y}\right)\) and \(\frac{\partial}{\partial\tilde{z}}=\frac{1}{2}\left(\frac{\partial}{\partial x }+i\frac{\partial}{\partial y}\right)\). Here, \(i\) represents the imaginary unit.

The Beltrami coefficient \(\mu:\Omega\rightarrow\mathbb{C}\) is a complex-valued function defined on the image domain \(\Omega\). It quantifies the local geometric distortion under the mapping \(g\). In particular, if \(\mu(z)=0\), then \(\frac{\partial f}{\partial z}(z)=0\), indicating that \(f\) is conformal at \(z\). According to quasiconformal theories, there exists a one-to-one correspondence between the space of homeomorphic mappings and the space of Beltrami coefficients. Given a homeomorphic deformation, its associated Beltrami coefficient \(\mu\) can be obtained using Equation 2. Conversely, given a Beltrami coefficient \(\mu\) with \(||\mu||_{\infty}<1\), the associated homeomorphic mapping \(f\) can be reconstructed by solving Beltrami's equation. In particular, the condition \(||\mu||_{\infty}<1\) ensures that the reconstructed mapping \(f\) is bijective. In our framework, we aim to generate a bijective deformation \(f\) to remove the geometric distortion while preserving the essential characteristics of the original image. This can be achieved by controlling the Beltrami coefficient, which measures the local geometric distortion of the associated mapping.

In our proposed model, the QCTN takes the input image \(\tilde{I}\) and outputs the Beltrami coefficient \(\mu\) associated with the desired deformation map \(f\). To ensure the bijectivity of \(f\), \(\mu\) is constrained to satisfy \(||\mu||_{\infty}<1\) by applying an activation function. The resulting \(\mu\) is then fed into another network that outputs the mapping \(f\) corresponding to \(\mu\). The mapping \(f\) is then used to remove the geometric distortion from \(\tilde{I}\) for further processing.

## IV Deformation-invariant Neural Network (DINN)

In this section, we provide a comprehensive explanation of our general framework, known as the Deformation-invariant Neural Network (DINN), designed to address imaging tasks on distorted images. Each component of DINN will be discussed in subsequent subsections.

### _Overall framework_

In this subsection, we provide an overview of the overall framework of the Deformation-invariant Neural Network (DINN), as depicted in Figure 2. DINN takes the distorted image as input, which is then processed by the Quasiconformal Transformer Network (QCTN). The QCTN, a lightweight neural network, outputs a deformation map aimed at removing the geometric distortion from the distorted image(s). The QCTN comprises two main components: (1) the Beltrami coefficient estimator

Fig. 2: The overall framework of the proposed Deformation-invariant Neural Network (DINN). This framework consists of three principal modules: the Beltrami Coefficient (BC) estimator, the Beltrami Solver network (BSNet), and the network dedicated to a specific downstream imaging task.
and (2) the Beltrami Solver Network (BSNet). The Beltrami coefficient estimator is responsible for computing the Beltrami coefficient \(\mu\) associated with the deformation map \(f\), which is used to rectify the geometric distortion in the distorted image \(\tilde{I}\). By applying \(f\), the distorted image is transformed into a less distorted version, denoted as \(I^{\prime}\), which aligns more closely with the distribution of clean, non-distorted images. The resulting \(I^{\prime}\) is subsequently fed into a downstream network tailored to the specific imaging task. The following subsection provides a detailed description of the Beltrami coefficient estimator, BSNet, and the loss function employed for network training.

### _Quasiconformal Transformer Network (QCTN)_

A crucial component in DINN is the incorporation of the Quasiconformal Transformer Network (QCTN). The distorted image is fed into the QCTN, which outputs a deformation map. This deformation map is then used to spatially transform the distorted image into one that is less distorted for further processing. To better control the geometric properties of the deformation map, we incorporate quasiconformality into the network. Specifically, the QCTN consists of two components: (1) the Beltrami coefficient estimator and (2) the Beltrami Solver network (BSNet).

The Beltrami coefficient estimator aims to estimate the Beltrami coefficient associated with the desired deformation map. The Beltrami coefficient measures the geometric distortion caused by the deformation map, allowing for easy control of the geometric properties of the deformation map. The BSNet then outputs the corresponding deformation map associated with the estimated Beltrami coefficient. In the following subsections, we will provide a detailed description of each component.

#### Ii-B1 Beltrami coefficient estimator

The main feature of the QCTN is the utilization of the Beltrami coefficient (BC) to represent the deformation map, as opposed to the conventional approach of using the vector field. The BC, denoted as \(\mu\), quantifies the local geometric distortion caused by the deformation map. Specifically, if the norm of \(\mu\), denoted as \(|\mu|\), is close to zero, it indicates that the geometric distortion under the associated deformation map is minimal. Therefore, a loss function can be designed to minimize \(|\mu|\) in certain regions, thereby reducing the local geometric distortion.

Moreover, for a bijective deformation, the BC satisfies the condition \(||\mu||_{\infty}<1\). In this work, generating a bijective deformation map is crucial for mitigating the geometric distortion in the distorted image. As mentioned earlier, bijectivity is essential to preserve the fundamental characteristics of the original image. By incorporating the BC into the network, bijectivity can be easily enforced by introducing an activation function that guarantees \(||\mu||_{\infty}<1\). In the discrete case, the image domain is triangulated, and each deformation mapping is treated as a piecewise linear function over each triangular face. The first derivatives of a piecewise linear function are constant on each triangular face. Consequently, the Beltrami coefficient can be regarded as a complex-valued function defined over each triangular face.

As shown in Figure 3, the Beltrami coefficient estimator is an encode-decoder network \(\mathcal{G}^{\theta}\) with trainable parameters \(\theta\), which takes distorted image \(\tilde{I}\) as input and outputs a BC \(\mu\in\mathbb{C}^{m}\) associated with a deformation map \(f\) for restoring the distorted image. Here, \(m\) is the number of triangular faces in the discretization of the image domain. The corresponding deformation map \(f\) associated with the output BC \(\mu\) can be obtained by the BSNet \(\mathcal{H}\), which will be described in the next subsection. This mapping \(f=\mathcal{H}(\mu)\) is then used to transform the distorted image \(\tilde{I}\) into a distortion-free one \(I^{\prime}=\tilde{I}\circ f\). In order to obtain a bijective deformation map, the BC \(\mu\) outputted by \(\mathcal{G}^{\theta}\) should satisfy the condition that \(||\mu||_{\infty}<1\). For this purpose, in the last layer of the Beltrami coefficient estimator, we apply the following activation function:

\[\mathcal{A}(\mu_{j})=\frac{e^{|\mu_{j}|}-e^{-|\mu_{j}|}}{e^{|\mu_{j}|}+e^{-| \mu_{j}|}}\textbf{arg}(\mu_{j}), \tag{3}\]

Fig. 3: The architecture of the Beltrami coefficient (BC) estimator.
where \(\mu_{j}\) is the \(j\)-th entry of \(\mu\in\mathcal{C}^{m}\) and \(\mathbf{arg}(\mu_{j})\) is the argument of the complex number \(\mu_{j}\).

The activation function \(\mathcal{A}\) ensures that the network outputs \(\mu\) whose supremum norm is strictly less than 1. The associated deformation map is then bijective. The parameters of \(\mathcal{G}^{\theta}\) are optimized by backward propagation to minimize suitable loss functions, which will be described in subsection IV-C.

#### Iv-B2 Beltrami Solver Network (BSNet)

Another component in the QCTN is the pretrained Beltrami Solver Network (BSNet). The BSNet, denoted by \(\mathcal{H}\), takes the BC \(\mu\in\mathbb{C}^{m}\) as input and outputs its corresponding deformation map \(f=\mathcal{H}(\mu)\). Mathematically, the BSNet solves Beltrami's equation:

\[\frac{\partial f}{\partial\bar{z}}=\mu\frac{\partial f}{\partial z}. \tag{4}\]

Beltrami's equation has a variational formulation and can be converted into a system of elliptic partial differential equations:

\[\nabla\cdot\left(A\begin{pmatrix}u_{x}\\ u_{y}\end{pmatrix}\right)=0;\ \ \nabla\cdot\left(A\begin{pmatrix}v_{x}\\ v_{y}\end{pmatrix}\right)=0, \tag{5}\]

where \(A=\begin{pmatrix}\alpha_{1}&\alpha_{2}\\ \alpha_{2}&\alpha_{3}\end{pmatrix}\), \(\alpha_{1}=\frac{(\rho-1)^{2}+\tau^{2}}{1-\rho^{2}-\tau^{2}}\), \(\alpha_{2}=\frac{-2\tau}{1-\rho^{2}-\tau^{2}}\), \(\alpha_{3}=\frac{(\rho+1)^{2}+\tau^{2}}{1-\rho^{2}-\tau^{2}}\), and \(\mu=\rho+i\tau\). In the discrete case, the image domain \(\Omega\) is discretized by a triangulation mesh, and \(f\) is piecewise linear on each triangular face. Suppose \(f=\mathbf{u}+i\mathbf{v}\), where \(\mathbf{u}\) and \(\mathbf{v}\) are the coordinate functions of \(f\) defined on every vertex. Then, the system of elliptic PDEs can be discretized into two sparse linear systems: \(C_{1}\mathbf{u}=\mathbf{0}\) and \(C_{1}\mathbf{v}=\mathbf{0}\). These can be used to define a loss function to train the BSNet:

\[\mathcal{L}_{BSNet}=||C_{1}\mathbf{u}||_{1}+||C_{2}\mathbf{v}||_{1}. \tag{6}\]

The architecture of the BSNet is shown in 4. The network consists of a short path (upper network) and a long path (bottom network). The goal is to design a smaller network with fewer parameters for efficient training. To achieve this goal, we consider the Fourier transform of \(\mu\). This is inspired by the property that the low-frequency component of \(\mu\) can effectively capture the overall pattern of the corresponding deformation map \(f\)[29]. In the long path, we first perform the discrete Fourier transform on \(\mu\), and then apply truncation to the Fourier coefficient matrix to keep a few coefficients associated with the low-frequency component. The truncated Fourier coefficient matrix contains the major information of the corresponding mapping. After that, the truncated Fourier coefficient matrix is fed into the Domain Transform Layer (DTL), which imitates the process of transforming features from the frequency domain to the spatial domain. The network is then extended with multiple convolutional layers, each followed by an activation function. The truncation of the Fourier coefficient matrix greatly reduces the number of variables and parameters in the network. However, some subtle information may be lost due to the truncation of high-frequency components of the Fourier coefficients. Specifically, some local deformation patterns may be lost. To address this, we add a short path, which consists of a few layers of convolution and downsampling. The output is concatenated with the output from the long path. This short path is shallow and has a minimal training burden on the overall network. More details about the BSNet can be found in [30].

### _Training process of DINN_

In the DINN framework, our main task is to train the Beltrami coefficient estimator and the BSNet. The Beltrami coefficient estimator is lightweight, making the training process cost-effective. The BSNet can either be pretrained or trained simultaneously,

Fig. 4: The architecture of the Beltrami Solver Network
depending on the application. The downstream network for performing a specific imaging task is pretrained using the clean, undistorted training dataset. For the ease of our discussion, denote the Beltrami coefficient estimator, BSNet, and the downstream network for the imaging task as \(\mathcal{N}_{\theta}\), \(\mathcal{H}_{\phi}\), and \(\mathcal{T}_{\varphi}\), respectively.

To train the DINN framework, we optimize the loss function with suitable loss terms associated with each component of the DINN. The loss function \(\mathcal{L}\) is in the form:

\[\mathcal{L}(\theta,\phi)=\alpha\mathcal{L}_{est}+\beta\mathcal{L}_{BSNet}+\gamma \mathcal{L}_{task}. \tag{7}\]

\(\mathcal{L}_{est}\) guides the training of the Beltrami coefficient estimator by enforcing it to output a Beltrami coefficient associated with a suitable deformation map, which aligns the deformed image with its ground truth. For instance, suppose the distorted images and their corresponding ground truth images are available in the training data. Let \(\tilde{I}\) be the distorted image and \(I\) be its ground truth. Then, \(\mathcal{L}_{est}\) can be designed to measure the mean square error between the deformed image and the ground truth image:

\[\mathcal{L}_{est}=||\tilde{I}\circ\mathcal{H}_{\phi}\circ\mathcal{N}_{\theta} (\tilde{I})-I||_{2}. \tag{8}\]

In some applications where the ground truth deformation map \(f_{\tilde{I}}\) to restore the distorted image \(\tilde{I}\) is known, \(\mathcal{L}_{est}\) can be designed as:

\[\mathcal{L}_{est}=||\mathcal{H}_{\phi}\circ\mathcal{N}_{\theta}(\tilde{I})-f_ {\tilde{I}}||_{2}. \tag{9}\]

Additionally, \(\mathcal{L}_{BSNet}\) guides the training of the parameters \(\phi\) of the BSNet, ensuring that \(\mathcal{H}\) solves Beltrami's equation. In practice, this network can also be pretrained, and we can set \(\beta=0\).

Finally, \(\mathcal{L}_{task}\) is the loss function used to train the downstream network for the specific imaging task. It is included in the loss \(\mathcal{L}\) to guide the output deformation map \(f=\mathcal{H}_{\phi}\circ\mathcal{N}_{\theta}(\tilde{I})\) such that the deformed image \(I^{\prime}=\tilde{I}\circ f\) lies within the distribution of clean, undistorted images. By adding \(\mathcal{L}_{task}\) to the loss \(\mathcal{L}\), we aim to find \(f\) such that the deformed image \(I^{\prime}\) produced by \(f\) gives an accurate imaging result when fed into the pretrained downstream network \(\mathcal{T}_{\varphi}\). Note that \(\mathcal{T}_{\varphi}\) is pretrained on the training dataset of undistorted images. Thus, minimizing \(\mathcal{L}_{task}\) encourages \(f\) to deform \(\tilde{I}\) to one that aligns with the training dataset of undistorted images. For example, for image classification, \(\mathcal{L}_{task}\) can be chosen as the cross-entropy of the probability vectors. If \(I^{\prime}\) remains distorted, \(\mathcal{T}_{\varphi}\) is likely to produce an incorrect probability vector, resulting in a large cross-entropy with the correct probability vector (the label). By minimizing the cross-entropy, we guide the Beltrami coefficient estimator \(\mathcal{N}_{\theta}\) to output a Beltrami coefficient associated with a deformation map \(f\) that restores the deformed image \(I^{\prime}\) to an image that aligns with the training dataset of undistorted images of its corresponding class. The cross-entropy between \(\mathcal{T}_{\varphi}(I^{\prime})\) and the ground truth probability vector will be small.

In the following section, we will demonstrate the application of the DINN framework to real imaging tasks in order to illustrate the concept of the framework.

## V Applications of DINN

In this section, we describe how we can apply the DINN framework to three real applications, namely, (1) image classification; (2) image restoration and (3) 1-1 facial verification.

### _Classification of distorted images_

The DINN framework can be applied to perform image classification on distorted images. In certain real-world scenarios, images can undergo geometric distortions. For example, images captured using a long-range camera may experience geometric distortions caused by atmospheric turbulence. When dealing with distorted images, the predictions made by a classification network \(\mathcal{T}_{c}\) can be inaccurate due to the mismatch between the distorted image \(\tilde{I}\) and the training dataset, which consists of clean and undistorted images. To tackle this issue, we utilize the DINN framework to develop a deep neural network specifically designed for classifying distorted images. The main idea is to incorporate the QCTN component before the downstream classification network. The overall network architecture is illustrated in Figure 5. Initially, a distorted image \(\tilde{I}\) is inputted into the QCTN, which then produces a deformation map \(f\). The deformed image \(I^{\prime}\) by \(f\) is subsequently fed into the pre-trained downstream classification network, resulting in a probability vector.

Fig. 5: DINN based image classification network.
Suppose we are given a training dataset of distorted images \(\tilde{I}\), whose classification labels are known. In certain situations, a training dataset may also be generated using physical experiments or mathematical simulations, resulting in paired images that comprise distorted images and their corresponding original counterparts. For example, one approach is to submerge a ground truth image in water and capture the image, accounting for water turbulence, to generate distorted images. This training dataset serves the purpose of guiding or initializing the parameters of the QCTN. To train the network, we optimize the parameters so that they minimize the following loss function:

\[\mathcal{L}_{c}=\alpha\mathcal{L}_{MSE}+\beta\mathcal{L}_{BSNet}+\gamma \mathcal{L}_{ce}, \tag{10}\]

where \(\mathcal{L}_{MSE}\) is the mean square error given by \(||\tilde{I}-I||_{2}^{2}\). \(\mathcal{L}_{ce}\) is the cross entropy loss given by \(\sum_{i=1}^{n}p_{i}\log q_{i}\), where \(\mathbf{p}=(p_{i})_{1\leq i\leq n}\) is the ground truth probability vector and \(\mathbf{q}=(q_{i})_{1\leq i\leq n}\) is the predicted probability obtained by the classification network \(\mathcal{T}_{c}\). In case the original undistorted images corresponding to the distorted images in the training dataset are unavailable, we can set \(\alpha_{1}=0\). The BSNet can be pretrained, in which case \(\alpha_{2}\) can be set to 0. As mentioned in the previous subsection, \(\mathcal{L}_{ce}\) guides the Beltrami coefficient estimator \(\mathcal{N}_{\theta}\) to produce a Beltrami coefficient associated with a deformation map \(f\). This deformation map is responsible for transforming the distorted image \(I^{\prime}\) into its undistorted version, aligning it with the distribution of clean, undistorted images.

### _Image restoration_

The DINN can be applied to image restoration for turbulence-distorted images. These images can be affected by atmospheric turbulence or water turbulence. Our proposed model leverages a GAN-based architecture, as depicted in Figure 6. The deep neural network comprises the QCTN and an image deblurring module. The image deblurring module aims to restore a blurry image. Besides, turbulence-distorted images exhibit geometric distortions. The QCTN aims to eliminate the geometric distortions. Initially, the geometrically distorted image \(\tilde{I}\) is inputted into the QCTN, which generates an appropriate deformation map. The resulting deformed image, denoted as \(I^{\prime}\), is an improved version with the geometric distortions eliminated. However, it is possible for \(I^{\prime}\) to suffer from blurring caused by the spatial resampling process. To address this, \(I^{\prime}\) is further processed by a color correction network \(\mathcal{N}_{cc}^{\theta}\), resulting in a color-corrected image referred to as \(I^{\prime\prime}\). By combining the outputs of the QCTN and CCnet, we obtain the restored image \(I^{\prime\prime}\), which serves as the generator within the GAN framework. Additionally, a discriminator \(D\) is trained to evaluate the quality of the generated image and provide feedback to the generator. Throughout the training process, the generator aims to enhance the quality of the generated image, while the discriminator strives to distinguish between the restored and undistorted images.

In this work, our training dataset consists of pairs of geometrically distorted images and their corresponding undistorted originals. To train the network, we optimize the parameters so that they minimize the following loss function:

\[\mathcal{L}=a_{1}\mathcal{L}_{MSE}(I,I^{{}^{\prime}})+a_{2} \mathcal{L}_{MSE}(I,I^{{}^{\prime\prime}})+a_{3}\mathcal{L}_{vgg}(I,I^{{}^{ \prime}}) \tag{11}\] \[+a_{4}\mathcal{L}_{vgg}(I,I^{{}^{\prime\prime}})+a_{5}\mathcal{L} _{adv}(I^{{}^{\prime\prime}}),\]

Here, \(\mathcal{L}_{MSE}\) is as defined in the last subsection. It is introduced in the loss function to guide the training of the QCTN as well as the CCnet. \(\mathcal{L}_{vgg}\) denotes the VGG loss, which measures the mean square error between the VGG features of two images:

\[\mathcal{L}_{vgg}(I_{1},I_{2})=||\Phi_{vgg}(I_{1})-\Phi_{vgg}(I_{2})||^{2}, \tag{12}\]

\(\Phi_{vgg}(I_{1})\) and \(\Phi_{vgg}(I_{2})\) denote the VGG features of \(I_{1}\) and \(I_{2}\) respectively. The VGGNet is pretrained on the ImageNet dataset for a classification task. The output of the last hidden layer is extracted and utilized as the VGG feature. Hence, the objective of the third and fourth terms is to encourage the VGG features of \(I^{\prime}\) and \(I^{\prime\prime}\) to closely resemble the VGG feature of the ground truth image \(I\). The last term \(\mathcal{L}_{adv}(I,I^{{}^{\prime\prime}})\) is the adversarial loss defined as:

\[\mathcal{L}_{adv}(I,I^{{}^{\prime\prime}})=\log(D(I))+\log(1-D(I^{\prime\prime} )), \tag{13}\]

Fig. 6: DINN based image restoration network.
where \(D\) is the discriminator for determining whether an input image is real or generated, yielding output values between 0 and 1, respectively. By minimizing the adversarial loss, the objective is to guide the synthesized image \(I^{\prime\prime}\) towards closely resembling a clean, undistorted image that the discriminator recognizes as real. In this task, the CCnet serves as a deblurring module and can be pretrained. The discriminator plays a crucial role in the min-max game of the GAN model and needs to participate in the training process. The optimization of the network parameters then follows an alternating approach, similar to the conventional GAN model. In this paper, we have set the weight parameters as \(a_{1}=1.0\), \(a_{2}=0.1\), \(a_{3}=0.5\), \(a_{4}=0.1\), and \(a_{5}=0.1\) for our experiments.

### _1-1 facial verification_

The DINN framework can also be applied to the problem of 1-1 facial verification. In this task, the objective is to determine whether a distorted facial image \(\tilde{I}^{x}\) belongs to the same person as another facial image \(I^{y}\). In real-world scenarios, facial images captured by long-range cameras often exhibit geometric distortion caused by air turbulence, particularly when considering zoomed-in images. Our proposed model is illustrated in Figure 7. Initially, the distorted image \(\tilde{I}^{x}\) is inputted into the image restoration network, as described in the previous subsection, resulting in a geometrically restored and color-corrected image \(I^{x}\). Subsequently, both the restored image \(I^{x}\) and the reference image \(I^{y}\) are fed into a feature extractor network, denoted as \(\mathcal{N}_{feature}\), which utilizes the _IR-50_ architecture [31]. The feature representations \(\mathcal{N}_{feature}(I^{x})\) and \(\mathcal{N}_{feature}(I^{y})\) are then passed to a similarity measure network, denoted as \(\mathcal{N}_{head}\), which employs the ArcFace similarity comparison method [32]. The final output of the model determines whether the two facial images belong to the same person, with a value of 1 indicating a match and 0 indicating a mismatch. During the training process, both \(\mathcal{N}_{feature}\) and \(\mathcal{N}_{head}\) are pre-trained. On the other hand, the image restoration network, which includes the QCTN, is trained according to the methodology described in the previous subsection.

## VI Experimental Results

In this section, we evaluate the efficacy of our proposed DINN framework through a series of experiments. Specifically, we assess the performance in image classification, image restoration, and 1-1 facial verification tasks involving distorted images using the DINN framework. We compare our results against those achieved by state-of-the-art methods. Additionally, we conduct self-ablation studies to explore the impact of various parameters and settings on the performance of the framework.

The experimental setting is described in detail below.

**Computational Resources and Parameters** To ensure a fair comparison, all models were trained using the RMSprop optimizer with a fixed learning rate of \(0.00001\). Each method underwent 100 epochs of optimization to achieve sufficient convergence. The batch size was set to 64, unless stated otherwise. The training process took place on a CentOS 8.1 central cluster computing node equipped with two Intel Xeon Gold 5220R 24-core CPUs and two NVIDIA V100 Tensor Core GPUs.

**Training Details** For the classification task, the classifier network is pre-trained and kept fixed. Subsequently, the QCTN module is trained using the loss function defined in Equation (10). In the image restoration and 1-1 facial verification tasks, the overall model consists of multiple components: the Beltrami coefficient Estimator, the Color Correction Network (CCnet), and the discriminator. The training process follows an alternating minimizing approach.

### _DINN for classification of distorted images_

In this subsection, we provide the experimental results of image classification for distorted images using the method introduced in subsection V-A. We evaluate the performance of the method on images distorted by different types of spatial deformations,

Fig. 7: DINN based 1-1 facial verification network.
specifically (1) affine transformations, (2) elastic transformations, and (3) a combination of affine and elastic transformations. The objective is to assess the capability of the proposed DINN framework in effectively handling various types of deformations.

**Affine Deformation** We evaluated the performance of our proposed method on the MNIST handwriting dataset distorted by affine transformations. To introduce these deformations, we applied rotation angles within the range of \([-\frac{\pi}{3},\frac{\pi}{3}]\) and scaling parameters within the range of \([0.2,0.6]\), while allowing translations within the image domain. The downstream image classification network utilized in our experiments is a convolutional neural network consisting of three convolutional layers and two fully connected layers. In our proposed model, the QCTN is appended before the downstream classification network to correct the geometric distortions. For comparison purposes, we also considered a related framework that employs the spatial transformer network (STN) [16]. In this alternative model, the STN is appended in front of the downstream classification network instead of the QCTN.

During the classification stage, models equipped with transformer layers demonstrated better performance in terms of classification accuracy on the test dataset compared to the baseline convolutional neural network. The classification accuracy was notably low when the transformer layer was omitted. However, a significant improvement in classification accuracy was observed when the STN was added before the classification network. Furthermore, the incorporation of the QCTN resulted in even higher classification accuracy. A summary of the results can be found in Table I. The models were trained for 100 epochs to ensure convergence.

**Elastic Deformation** We assessed the performance of our proposed method on images distorted by general elastic spatial deformations. Elastic deformations are commonly encountered in various scenarios, such as captur

Fig. 8: Images produced by the deformation maps from different transformer layers. (a) Distorted images. (b) Visualization of the mapping generated by TPS-STN on a deformed image. (c) Image recovered by TPS-STN. (d) Visualization of the mapping generated by QCTN on a deformed image. (e) Image recovered by QCTN. The class names in the top-right corner of images in columns (a), (c), and (e) indicate the predicted class by the baseline CNN, TPS-STN, and DINN, respectively. The correct labels from the top row to the last row are Bird, Deer, Deer and Truck.

\begin{table}
\begin{tabular}{c|c|c c c} \hline \hline \multirow{2}{*}{\begin{tabular}{c} Deform Type \\ \end{tabular} } & Method & Train ACC & Test ACC & Invertible \\ \hline \multirow{2}{*}{Affine \&} & CNN & 91.62 & 82.73 & \multirow{2}{*}{\begin{tabular}{c} \\ \end{tabular} } \\  & ST-CNN & 97.97 & 94.90 & \\  & DINN-CNN & 97.45 & **96.32** & \\ \hline \multirow{2}{*}{Elastic} & CNN & 95.43 & 78.47 & \multirow{2}{*}{\begin{tabular}{c} \\ \end{tabular} } \\  & TPS-CNN & 99.37 & 81.94 & \\  & DINN-CNN & 99.11 & **84.58** & \\ \hline \multirow{2}{*}{Affine \&} & CNN & 76.77 & 70.29 & \multirow{2}{*}{
\begin{tabular}{c} \\ \end{tabular} } \\  & ST-CNN & 81.65 & 77.21 & \\ \hline \multirow{2}{*}{Elastic} & TPS-CNN & 86.13 & 80.63 & \\  & DINN-CNN & 86.48 & **83.06** & \\ \hline \hline \end{tabular}
\end{table} TABLE I: The results for classifying standard CIFAR-10 by different methods.
surfaces like glasses or water. To test the effectiveness of our method, we conducted experiments on the CIFAR10 dataset with large elastic deformations. Figure 8 shows some examples of the deformed images. For this experiment, we employed the deep layer aggregation model (DLA) [33] as our downstream classification network. To ensure a fair comparison, we implemented a variant of the spatial transformer network (STN) with thin-plate spline transformation, called the _TPS-STN_. This variant outputs the mapped coordinates of the control points without constraint. The architecture for the TPS-STN that predicts the mapped coordinates is the same as our Beltrami coefficient estimator. The primary distinction between QCTN and TPS-STN is that QCTN generates a bijective folding-free deformation, whereas TPS-STN does not possess this property. The bijectivity of QCTN plays a crucial role in this imaging task.

Table I presents the classification accuracy results of the downstream classification network, the classification network with TPS-STN, and the classification network with QCTN. Again, the classification models equipped with the transformer network exhibit significantly better accuracy. However, the use of QCTN yielded better results compared to TPS-STN. This is attributed to the non-bijective nature of the deformations produced by TPS-STN, which have impacted its performance.

Figure 8 displays several examples of deformed images generated by the deformation maps produced by the transformer layer. The aim is to ensure that the deformed images effectively alleviate the geometric distortions present in the input distorted images. The results demonstrate that the images restored by the DINN approach are notably more accurate and closely resemble an image from their respective classes. This improved restoration aids the classification network in better identifying the class to which each image belongs. The top left corner label for each image indicates the class recognized by the classification network. It is observed that both the downstream classification network and the network with the inclusion of STN provided incorrect classifications. However, with the integration of QCTN, the classifications were accurate.

**Combined Deformation** We further evaluate the performance of our proposed model on images distorted by a combination of elastic and affine deformations, which generally involve large deformations. The experiment is conducted on the FashionMNIST dataset, utilizing the same downstream classification network as in the experiments on images distorted by affine transformations. We compare our method with the downstream classification network, the STN network, and the TPS-STN network. The classification accuracies are presented in Table I. Once again, the classification models equipped with the transformer network demonstrate significantly improved accuracy. However, the use of QCTN yields notably superior results compared to both STN and TPS-STN. Even for such large deformations, our proposed model successfully preserves the bijectivity of the deformation map, whereas TPS-STN fails to do so.

Figure 9 shows several examples of deformed images generated by the deformation maps produced by the transformer layer. Each image's top-left corner label indicates the class recognized by the classification network. The results demonstrate that the images restored by the DINN approach are notably more accurate and closely resemble an image from their respective classes, even with such large deformations. It is observed that both the downstream classification network, the network with the inclusion of STN and the network with the inclusion of TPS-STN, provided incorrect classifications. On the other hand, the classifications were accurate under the DINN framework.

### _Restortion of Turbulence-distorted images_

In this subsection, we present the experimental results of image restoration for turbulence-distorted images using our proposed model, as introduced in subsection V-B. Turbulence-distorted images commonly occur when imaging through turbulent refractive

Fig. 9: Images produced by the deformation maps from different transformer layers. (a) Distorted images from FashionMNIST. (b) Visualization of the mapping generated by STN on a deformed image. (c) Image recovered by STN. (d) Visualization of the mapping generated by TPS-STN on a deformed image. (e) Image recovered by TPS-STN. (f) Visualization of the mapping generated by QCTN on a deformed image. (g) Image recovered by QCTN. The class names in the top-right corner of images in columns (a), (c), (e) and (g) indicate the predicted class by the baseline CNN, STN, TPS-STN, and DINN, respectively. The correct labels from the top row to the last row are Shirt and Scandal.
media, such as air and water, owing to the refraction and scattering of light [34]. These distortions pose significant challenges in achieving high-quality and undistorted images for further analysis.

In our experiments, we obtained the training dataset through the following process. To simulate air-turbulence distortion in images, we utilized the model proposed by [35]. This model requires specific parameters related to the virtual camera. For our experiment, we set the focal distance of the virtual camera to \(300mm\), with a lens diameter of \(5.357cm\) and a pixel size of \(4\times 10^{3}mm\). The virtual camera was positioned at an elevation of \(4m\) with an object distance of \(2km\). For weak turbulence, we set the turbulence strength parameter \(C_{n}^{2}=3.6\times 10^{-13}\), and for strong turbulence, we used \(C_{n}^{2}=3.6\times 10^{-12}\). By applying the turbulence fields obtained from the simulated turbulence model, we distorted the IMAGENET dataset, resulting in a training dataset comprising air-turbulence distorted images. To create the water turbulence image dataset, we utilized a physics-based ray tracer following the methodology described in [25]. This approach allowed us to simulate various types of waves for realistic water deformations. In our experiment, we specifically employed two water deformation types: _Ripple_ and _Ocean_. Some examples of the turbulence fields are shown in Figure 11.

In all of our experiments, we utilized a total of \(360,000\) images for training purposes. Additionally, we reserved a separate set of \(40,000\) images specifically for testing and evaluation.

Fig. 11: Examples of turbulence fields. The Left shows the turbulence field of the air turbulence. The right shows the turbulence field of the water turbulence.

Fig. 10: Results of image restoration for images corrupted by ocean type water turbulence by different methods.
We compare our proposed DINN-GAN model with other state-of-the-art methods for the restoration of distorted images, namely Pix2Pix[36], DeblurGAN[20], CycleGAN[22], LiGAN[26], and DTD-GAN[27]. Figure 10 presents the image restoration results for images distorted by ocean turbulence. Our proposed method achieves the best results by successfully removing geometric distortions. In contrast, the restored images produced by other methods still exhibit some turbulence distortions. Similarly, Figure 12 displays the image restoration results for images distorted by air turbulence, where, once again, our method outperforms the other approaches. Table II provides a quantitative comparison among the different methods, further demonstrating that our method delivers the best results.

Furthermore, we assessed the performance of the DINN-GAN model on real images captured by a digital camera, capturing scenes inside a pool where the images were distorted due to water flow. The distorted images are presented in Figure 13 (left), while Figure 13 (right) displays the restored image using the DINN-GAN model. Our model adeptly restores the images by effectively removing the geometric distortions. This once again highlights the effectiveness of our proposed method.
proposed by [35]. This experiment used the same set of parameters as in the previous subsection, with the turbulence strength parameter set to \(C_{n}^{2}=3.6\times 10^{-13}\). We utilized 450 subjects for training and reserved the remaining 50 subjects for testing.

Figure 14 showcases some ground truth facial images in the first column, while the last column displays the same images distorted by strong air turbulence. The figure also presents the restored images produced by our proposed method, along with p2pGAN, DeblurGAN, CycleGAN, LiGAN, and DTD-GAN. Additionally, for each example, corresponding error maps are included, which are defined as the mean square error between the predicted and ground truth values at each pixels. In these error maps, red indicates higher error, while blue represents lower error. Evidently, the restored images using the DINN framework exhibit the most favorable results. This observation is further supported by the quantitative comparison presented in Table III, which also includes the accuracy of 1-1 facial verification using different methods. Once again, our method achieves significantly higher accuracy compared to the other methods.

### _Self Ablation_

The performance of the proposed Deformation-Invariant Neural Network (DINN) as a learning approach may rely on the chosen network architecture. It is crucial to strike a balance between the depth of the architecture and its convergence capabilities. An architecture that is too shallow may fail to converge adequately, resulting in suboptimal performance. On the other hand,

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline Depth of Net & 2 & 3 & 4 & 5 \\ \hline PSNR & 20.2672 & 21.4485 & 21.6674 & 21.6931 \\ SSIM & 0.5325 & 0.5977 & 0.6134 & 0.6348 \\ MSE & 0.0339 & 0.0294 & 0.0281 & 0.273 \\ \hline \hline Conv. Type & Quadruple & Triple & Double & Single \\ \hline PSNR & 21.8411 & 21.7485 & 21.4485 & 20.0812 \\ SSIM & 0.6353 & 0.6163 & 0.5977 & 0.5184 \\ MSE & 0.0252 & 0.0268 & 0.0294 & 0.0326 \\ \hline \hline \end{tabular}
\end{table} TABLE IV: Self-ablation.

Fig. 13: Image restoration results for real images distorted by water turbulence.

\begin{table}
\begin{tabular}{c|c c c c c c c} \hline \hline Metrics & Sharp & Distorted & Pix2Pix & DeblurGAN & CycleGAN & Li _et al._ & Rai _et al._ & DINN-GAN \\ \hline PSNR & / & 21.6473 & 22.1846 & 24.4311 & 24.3958 & 25.3578 & 25.5666 & **26.3400** \\ SSIM & / & 0.6738 & 0.6185 & 0.7126 & 0.7129 & 0.7931 & 0.8015 & **0.8604** \\ MSE & / & 0.0097 & 0.0077 & 0.0081 & 0.0084 & 0.0067 & 0.0065 & **0.0063** \\ Accuracy & 95.31 & 81.23 & 83.58 & 85.08 & 84.98 & 86.76 & 88.53 & **90.15** \\ \hline \hline \end{tabular}
\end{table} TABLE III: Recognition accuracy and image quality evaluation on distorted human face.
an excessively deep architecture may lead to excessive training consumption. To determine the optimal configuration, we conduct two self-ablation studies within our model. Specifically, we investigate the influence of various downsample levels and convolution layers on the performance of the DINN-based image restoration model. This analysis allows us to gain insights into the impact of these factors on the overall effectiveness of the DINN framework.

**Influence of Downsample Levels** The downsampled levels in the encoder-decoder architecture, utilized by the BC estimator, play a vital role in our model. To assess the influence of the number of downsampling levels on the generalization ability of our model, we conducted a self-ablation study. The results presented in Table IV compellingly demonstrate that a UNet architecture with a downsample level of 3 is sufficient to achieve effective learning and generate high-quality restoration mappings. This finding highlights the importance of striking a balance between the complexity of the architecture and its performance.

**Influence of Convolution Layers**

The number of convolution layers in each level of the encoder-decoder architecture is another crucial factor that affects the restoration quality. The results presented in Table IV indicate that employing a double-convolution configuration for each level in the encoder-decoder architecture is highly effective in producing satisfactory restoration results. This finding highlights the importance of carefully considering the number of convolution layers at each level to achieve optimal performance in the restoration process.

## VII Conclusion and Future Work

In this paper, we have introduced the deformation-invariant neural network (DINN) framework to solve the challenging problem of imaging tasks involving geometrically distorted images. Our proposed framework, incorporating the quasiconformal transformer network (QCTN), has demonstrated its effectiveness in addressing various imaging tasks, including image classification of

Fig. 14: Image restoration results for images severely distorted by strong air turbulence using different methods. The corresponding error maps, defined as the mean square error between the predicted and ground truth values at each pixel point, are shown below the restored images. In the error maps, red indicates higher error, while blue represents lower error.
distorted images, image restoration in the presence of atmospheric or water turbulence and 1-1 facial verification under strong air turbulence.

The key contributions of our work include the development of DINN, which ensures consistent latent features for geometrically distorted images capturing the same underlying object or scene. We have introduced the portable QCTN component, which allows large pretrained networks to process heavily distorted images without requiring additional tuning, thereby reducing computational costs. The QCTN generates bijective deformation maps that preserve the salient features of the original images, resulting in more accurate restoration and recognition results. Our experimental results have shown that the proposed DINN framework outperforms existing GAN-based restoration methods in scenarios involving atmospheric turbulence and water turbulence. Furthermore, the application of DINN to 1-1 facial verification under strong air turbulence has demonstrated its efficacy in enhancing the accuracy of facial recognition even in adverse conditions.

While our proposed framework has yielded promising results, there are still several avenues for future research. One potential direction is to investigate the application of the DINN framework to other imaging tasks, such as image registration and image segmentation. Additionally, it is worth noting that the current DINN framework may yield less satisfactory outcomes when confronted with very extreme deformations. Therefore, further exploration is needed to enhance the ability of the proposed model to handle such challenging scenarios.

In conclusion, the proposed DINN framework, incorporating the QCTN component, offers a powerful solution for addressing imaging tasks involving geometrically distorted images. Our experimental results have demonstrated its superiority in image classification, image restoration and facial verification tasks under challenging conditions. The DINN framework opens up new possibilities for handling geometric distortions in various applications and provides a valuable contribution to the field of deep learning in imaging and computer vision.
