Complex-Valued Neural Networks for Data-Driven Signal Processing and
  Signal Understanding
Complex-valued neural networks have emerged boasting superior modeling
performance for many tasks across the signal processing, sensing, and
communications arenas. However, developing complex-valued models currently
demands development of basic deep learning operations, such as linear or
convolution layers, as modern deep learning frameworks like PyTorch and Tensor
flow do not adequately support complex-valued neural networks. This paper
overviews a package built on PyTorch with the intention of implementing
light-weight interfaces for common complex-valued neural network operations and
architectures. Similar to natural language understanding (NLU), which as
recently made tremendous leaps towards text-based intelligence, RF Signal
Understanding (RFSU) is a promising field extending conventional signal
processing algorithms using a hybrid approach of signal mechanics-based insight
with data-driven modeling power. Notably, we include efficient implementations
for linear, convolution, and attention modules in addition to activation
functions and normalization layers such as batchnorm and layernorm.
Additionally, we include efficient implementations of manifold-based
complex-valued neural network layers that have shown tremendous promise but
remain relatively unexplored in many research contexts. Although there is an
emphasis on 1-D data tensors, due to a focus on signal processing,
communications, and radar data, many of the routines are implemented for 2-D
and 3-D data as well. Specifically, the proposed approach offers a useful set
of tools and documentation for data-driven signal processing research and
practical implementation.
Josiah W. Smith
# Complex-Valued Neural Networks for Data-Driven Signal Processing and Signal Understanding

###### Abstract

Complex-valued neural networks have emerged boasting superior modeling performance for many tasks across the signal processing, sensing, and communications arenas. However, developing complex-valued models currently demands development of basic deep learning operations, such as linear or convolution layers, as modern deep learning frameworks like PyTorch and Tensor flow do not adequately support complex-valued neural networks. This paper overviews a package built on PyTorch with the intention of implementing light-weight interfaces for common complex-valued neural network operations and architectures. Similar to natural language understanding (NLU), which as recently made tremendous leaps towards text-based intelligence, _RF Signal Understanding (RFSU)_ is a promising field extending conventional signal processing algorithms using a hybrid approach of signal mechanics-based insight with data-driven modeling power. Notably, we include efficient implementations for linear, convolution, and attention modules in addition to activation functions and normalization layers such as batchnorm and layernorm. Additionally, we include efficient implementations of manifold-based complex-valued neural network layers that have shown tremendous promise but remain relatively unexplored in many research contexts. Although there is an emphasis on 1-D data tensors, due to a focus on signal processing, communications, and radar data, many of the routines are implemented for 2-D and 3-D data as well. Specifically, the proposed approach offers a useful set of tools and documentation for data-driven signal processing research and practical implementation.

Complex-Valued Neural Networks (CVNNs) Deep Learning Artificial Intelligence Signal Processing Radar Synthetic Aperture Radar

## 1 Introduction

Although much work as already been done for complex-valued neural networks (CVNNs), starting back in the 1990s [1], many common operations for complex-valued tensors remain unsupported by modern deep learning frameworks like PyTorch and TensorFlow. In this paper, we introduce a lightweight wrapper built on PyTorch with two main objectives as follows

* Provide an efficient interface for complex-valued deep learning using PyTorch.
* Provide open-source implementations and documentation for complex-valued operation such activation functions, normalization layers, and attention modules to increase the research rate for CVNNs.

Fundamentally, complex-valued data have two degrees of freedom (DOF), which are commonly modeled as real and imaginary parts or magnitude and phase as

\[\mathbf{z}=\mathbf{x}+j\mathbf{y}=|\mathbf{z}|e^{j\angle\mathbf{z}}, \tag{1}\]

where \(j=\sqrt{-1}\) is known as the complex unit, \(\mathbf{x}\) and \(\mathbf{y}\) are the real and imaginary parts of \(\mathbf{z}\), and \(|\mathbf{z}|\) and \(\angle\mathbf{z}\) are the magnitude and phase of \(\mathbf{z}\).
### Existing CVNN Work

Recent surveys of CVNNs and their history can be found in [2, 3]. Starting in 2018, Trabelsi _et al._ introduced _deep complex networks_ (DCN) extending many common deep learning approaches to complex-valued data [4]. As mentioned in [5], some attempts have been made by PyTorch and TensorFlow to incorporate complex-values into the core architecture; however, much work remains to be done. The authors of [5] implement a CVNN framework in TensorFlow, with impressive functionality. However, recent studies have shown a drastic decline in TensorFlow usage among engineers, while PyTorch dominates industry and academia. The code publicized as part of this work was used extensively throughout the dissertation [6] in addition to [7, 8, 9, 10, 11, 12, 13, 14, 15].

One paramount issue for CVNNs is complex-valued backpropagation and computation of complex-valued gradients. Mathematical investigations for Liouville Theorem and Wirtinger Calculus can be found in [5, 4], and are omitted here for brevity. Whereas, [5] develops Wirtinger-based proper backpropagation for TensorFlow, PyTorch natively supports complex-valued backpropagation.

### Installation Notes

**IMPORTANT:** Prior to installation, install PyTorch to your environment using your preferred method using the compute platform (CPU/GPU) settings for your machine. PyTorch will not be automatically installed with the installation of complextorch and MUST be installed manually by the user.

#### 1.2.1 Install using PyPI

complextorch is available on the Python Package Index (PyPI) and can be installed using the following command:

pip install complextorch

#### 1.2.2 Install using GitHub

Useful if you want to modify the source code.

git clone [https://github.com/josiahwsmith10/complextorch.git](https://github.com/josiahwsmith10/complextorch.git)

## 2 Complex-Valued Layers

In this section, we overview the complex-valued layers supported by complextorch, which can be found at [https://complextorch.readthedocs.io/en/latest/nn/modules.html](https://complextorch.readthedocs.io/en/latest/nn/modules.html). Nearly all modules introduced in this section closely follow the PyTorch to improve integration and streamline user experience (UX) for new users.

### Gauss' Multiplication Trick

As pointed out in [16, 17, 18], many complex-valued operations can be more efficiently implemented by leveraging Gauss' multiplication trick. Suppose \(\mathcal{L}(\cdot)=\mathcal{Z}_{\mathbb{R}}(\cdot)+j\mathcal{Z}_{\mathbb{I}}(\cdot)\) is a linear operator, such as multiplication or convolution, and \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\). Hence,

\[\mathcal{L}(\mathbf{z})=\mathcal{L}(\mathbf{x})+j\mathcal{L}(\mathbf{y})= \mathcal{L}_{\mathbb{R}}(\mathbf{x})-\mathcal{L}_{\mathbb{I}}(\mathbf{y})+j \left(\mathcal{L}_{\mathbb{R}}(\mathbf{y})+\mathcal{L}_{\mathbb{I}}(\mathbf{ x})\right). \tag{2}\]

This is the common implementation of complex-valued operations in deep learning applications. However, it requires four computations, such as multiplications or convolutions, which can become computationally costly. Gauss' trick reduces the number of computations down to 3 as

\[t_{1} \triangleq\mathcal{Z}_{\mathbb{R}}(\mathbf{x}),\] \[t_{2} \triangleq\mathcal{L}_{\mathbb{I}}(\mathbf{y}), \tag{3}\] \[t_{3} \triangleq(\mathcal{Z}_{\mathbb{R}}+\mathcal{L}_{\mathbb{I}})( \mathbf{x}+\mathbf{y}),\] \[\mathcal{L}(\mathbf{z})=t_{1}-t_{2}+j(t_{3}-t_{2}-t_{1}). \tag{4}\]

This technique is leveraged throughout complextorch to improve computational efficiency whenever applicable.
### Complex-Valued Linear Layers

We extend linear layers, the backbone of perceptron neural networks, for CVNNs. Similar to Section 2.1, we define the complex-valued linear layer \(\mathsf{CVLinear}\) as

\[H_{\mathsf{CVLinear}}(\cdot)=H_{\mathsf{CVLinear}\cdot\mathbb{R}}(\cdot)+jH_{ \mathsf{CVLinear}\cdot\mathbb{I}}(\cdot), \tag{5}\]

where \(H_{\mathsf{CVLinear}\cdot\mathbb{R}}(\cdot)\) and \(H_{\mathsf{CVLinear}\cdot\mathbb{I}}(\cdot)\) can be implemented in PyTorch as real-valued Linear layers, as detailed in Table 1. Using (4), the linear layer can be efficiently computed with the composition \(H_{\mathsf{CVLinear}\cdot\mathbb{R}}(\cdot)+H_{\mathsf{CVLinear}\cdot \mathbb{I}}(\cdot)\) being a linear layer with the weights and bias of \(H_{\mathsf{CVLinear}\cdot\mathbb{R}}(\cdot)\) and \(H_{\mathsf{CVLinear}\cdot\mathbb{I}}(\cdot)\) summed.

### Complex-Valued Convolution Layers

Similarly, we define a general complex-valued convolution layer \(\mathsf{CVConv}\) as

\[H_{\mathsf{CVConv}}(\cdot)=H_{\mathsf{CVConv}\cdot\mathbb{R}}(\cdot)+jH_{ \mathsf{CVConv}\cdot\mathbb{I}}(\cdot), \tag{6}\]

where \(H_{\mathsf{CVConv}\cdot\mathbb{R}}(\cdot)\) and \(H_{\mathsf{CVConv}\cdot\mathbb{I}}(\cdot)\) can be implemented in PyTorch as various real-valued convolution layers, as detailed in Table 1. Using (4), the convolution layers layer can be efficiently computed with the composition \(H_{\mathsf{CVConv}\cdot\mathbb{R}}(\cdot)+H_{\mathsf{CVConv}\cdot\mathbb{I}}(\cdot)\) being a convolution layer with the kernel weights and bias of \(H_{\mathsf{CVConv}\cdot\mathbb{R}}(\cdot)\) and \(H_{\mathsf{CVConv}\cdot\mathbb{I}}(\cdot)\) summed.

### Complex-Valued Attention Layers

Whereas attention-based models, such as transformers, have gained significant attention for natural language processing (NLP) and image processing, their potential for implementation in complex-valued problems such as signal processing remains relatively untapped. Here, we include complex-valued variants of several attention-based techniques.

#### 2.4.1 Complex-Valued Scaled Dot-Product Attention

The ever-popular scaled dot-product attention is the backbone of many attention-based methods [19], most notably the transformer [20, 21, 22, 23, 24, 11].

Given complex-valued _query, key, and value_ tensors \(Q,K,V\), the complex-valued scaled dot-product attention can be computed as

\[\text{Attention}(Q,K,V)=\mathcal{S}(QK^{T}/t)V, \tag{7}\]

where \(t\) is known as the temperature (typically \(t=\sqrt{d_{attn}}\)) and \(\mathcal{S}\) is the softmax function.

It is important to note that unlike real-valued scaled dot-product attention, the complex-valued version detailed above must employ a complex-valued version of the soft-max function as the real-valued softmax is unsuited for complex-valued data. We implemented several complex-valued softmax function options detailed in Section 2.5.

Additionally, we include an implementation of multi-head attention, which is commonly employed in transformer and attention-based models [22].

\begin{table}
\begin{tabular}{c|c} complextorch & PyTorch \\ \hline \(\mathsf{CVConv}\) & Linear \\ \end{tabular}
\end{table}
Table 1: PyTorch equivalent of the complex-valued linear layer.

\begin{table}
\begin{tabular}{c|c} complextorch & PyTorch \\ \hline \(\mathsf{CVConv}\) & Conv1d \\ \(\mathsf{CVConv}\) & Conv2d \\ \(\mathsf{CVConv}\) & Conv3d \\ \(\mathsf{CVConv}\) & Conv1d \\ \(\mathsf{CVConv}\) & Conv1d \\ \(\mathsf{CVConv}\) & Conv1d \\ \(\mathsf{CVConv}\) & Conv1d \\ \(\mathsf{CVConv}\) & Conv1d \\ \(\mathsf{CVConv}\) & Conv1d \\ \end{tabular}
\end{table}
Table 2: PyTorch equivalent of complex-valued convolution layers.
#### 2.4.2 Complex-Valued Efficient Channel Attention (CV-ECA)

Efficient Channel Attention (ECA) was first introduced in [25]. Here, we extend ECA to complex-valued data for CV-ECA. Following the construction of ECA, we define CV-ECA as

\[\texttt{CV-ECA}=\mathcal{M}\left(H_{\texttt{CVConv1d}}(H_{\texttt{CVModuleViewAvgPoolId}}( \mathbf{z}))\right)\odot\mathbf{z}, \tag{8}\]

where \(\mathcal{M}(\cdot)\) is the masking function (some options are implemented in Sections 2.5 and 2.6), \(H_{\texttt{CVConv1d}}(\cdot)\) is the 1-D complex-valued convolution layer defined in Section 2.3, and \(H_{\texttt{CVAdaptiveAvgPoolId}}(\cdot)\) is the complex-valued global adaptive average pooling layer for the \(N\)-D input tensor detailed in Section 2.11. It is important to note that the 1-D convolution is computed along the channel dimension of pooled data. This is the notable difference between ECA and MCA.

#### 2.4.3 Complex-Valued Masked Channel Attention (CV-MCA)

Complex-valued masked channel attention (CV-MCA) is similar to CV-ECA but employs a slightly different implementation. Generally, the masked attention module implements the following operation

\[\texttt{CV-MCA}(\mathbf{z})=\mathcal{M}(H_{\texttt{ConvUp}}(\mathcal{A}(H_{ \texttt{ConvDown}}(\mathbf{z}))))\odot\mathbf{z}, \tag{9}\]

where \(\mathcal{M}(\cdot)\) is the masking function (some options are implemented in Sections 2.5 and 2.6), \(H_{\texttt{ConvUp}}(\cdot)\) and \(H_{\texttt{ConvDown}}(\cdot)\) are \(N\)-D convolution layers with kernel sizes of 1 that reduce the channel dimension by a factor \(r\), and \(\mathcal{A}(\cdot)\) is the complex-valued non-linear activation layer (see Section 2.8). The implementations of complex-valued masked attention are included for 1-D, 2-D, and 3-D data. For more information on complex-valued masked channel attention, see the paper that introduced it [26].

### Complex-Valued Softmax Layers

Softmax is an essential function for several tasks throughout deep learning. Complex-valued softmax functions have not been explored thoroughly in the literature at the time of this paper. However, a similar route, know as masking, has seen some attention in recent research [26].

Here, we introduce several softmax layer suitable for complex-valued tensors.

#### 2.5.1 Split Type-A Complex-Valued Softmax Layer

Using the definition of a split Type-A function from Section 2.8.1, we define the complex-valued split Type-A softmax layer (\(\texttt{CVSoftMax}\)) as

\[\texttt{CVSoftMax}(\mathbf{z})=\texttt{SoftMax}(\mathbf{x})+j\texttt{SoftMax }(\mathbf{y}), \tag{10}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\) and \(\texttt{SoftMax}\) is the PyTorch real-valued softmax function.

#### 2.5.2 Phase Preserving Softmax

Similar to a Type-B function from Section 2.8.2, we define the complex-valued phase preserving softmax layer (PhaseSoftmax) as

\[\texttt{CVSoftMax}(\mathbf{z})=\texttt{SoftMax}(|\mathbf{z}|)\odot\frac{ \mathbf{z}}{|\mathbf{z}|}, \tag{11}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\) and \(\texttt{SoftMax}\) is the PyTorch real-valued softmax function.

#### 2.5.3 Magnitude Softmax

We define the magnitude softmax layer, which simply computes the softmax over the magnitude of the input and ignores the phase, (\(\texttt{MagSoftmax}\)) as

\[\texttt{CVSoftMax}(\mathbf{z})=\texttt{SoftMax}(|\mathbf{z}|), \tag{12}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\) and \(\texttt{SoftMax}\) is the PyTorch real-valued softmax function.
### Complex-Valued Masking Layers

#### 2.6.1 Complex Ratio Mask (cRM) or Phase Preserving Sigmoid

Detailed in Eq. (23) of [26], the complex ratio mask (cRM) applies the traditional sigmoid function to the magnitude of the signal while leaving the phase information unchanged as

\[\texttt{ComplexRatioMask}(\mathbf{z})=\texttt{Sigmoid}(|\mathbf{z}|)\odot \frac{\mathbf{z}}{|\mathbf{z}|} \tag{13}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\) and Sigmoid is the PyTorch real-valued softmax function.

#### 2.6.2 Magnitude Min-Max Normalization Layer

The min-max norm, which has proven useful in complex-value data normalization [13], is another option for a complex-valued softmax function as

\[\texttt{MagMinMaxNorm}(\mathbf{z})=\frac{\mathbf{z}-\mathbf{z}_{\text{min}}}{ \mathbf{z}_{\text{max}}-\mathbf{z}_{\text{min}}} \tag{14}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\) and \(\mathbf{z}_{\text{min}}\) and \(\mathbf{z}_{\text{max}}\) indicate the minimum and maximum of \(|\mathbf{z}|\).

### Complex-Valued Normalization Layers

Normalization layers are a crucial aspect of modern deep learning algorithms facilitating improved convergence and model robustness. As discussed in [4], traditional standardization of complex-valued data is not sufficient to translate and scale the data to unit variance and zero mean. Rather, a whitening procedure is necessary to ensure a circular distribution with equal variance for the real and imaginary parts of the signal. The whitening algorithm is derived in greater detail in [4], but we employ the same procedure for both batch normalization and layer normalization.

### Complex-Valued Activation Layers

Complex-valued activation functions are a key element of CVNNs that different significantly from real-valued neural networks. The popular real-valued activation layers (such as ReLU and GeLU) cannot be directly applied to complex-valued data without some modification. Complex-valued activation functions must take into account the 2 degrees-of-freedom inherent to complex-valued data, typically represented as real and imaginary parts or magnitude and phase.

We highlight four categories of complex-valued activation layers:

* Split Type-A Activation Layers
* Split Type-B Activation Layers
* Fully Complex Activation Layers
* ReLU-Based Complex-Valued Activation Layers

Split _Type-A_ and _Type-B_ activation layers apply real-valued activation functions to either the real and imaginary or magnitude and phase, respectively, of the input signal [2, 5]. Fully complex activation layers are entirely complex-valued, while ReLU-based complex-valued activation layers are the family of complex-valued activation functions that extend the ever-popular ReLU to the complex plane.

#### 2.8.1 Split Type-A Complex-Valued Activation Layers

_Type-A_ activation functions consist of two real-valued functions, \(G_{\mathbb{R}}(\cdot)\) and \(G_{\mathbb{I}}(\cdot)\), which are applied to the real and imaginary parts of the input tensor, respectively, as

\[G(\mathbf{z})=G_{\mathbb{R}}(\mathbf{x})+jG_{\mathbb{I}}(\mathbf{y}). \tag{15}\]

In most cases, \(G_{\mathbb{R}}(\cdot)=G_{\mathbb{I}}(\cdot)\); however, \(G_{\mathbb{R}}(\cdot)\) and \(G_{\mathbb{I}}(\cdot)\) can also be distinct functions. Table 3 details the Type-A activation functions included in complextorch. Additionally, a generalized Type-A activation function is included allowing the user to adopt any set of \(G_{\mathbb{R}}(\cdot)\) and \(G_{\mathbb{I}}(\cdot)\) desired.
#### 2.8.2 Polar Type-B Complex-Valued Activation Layers

Similarly, _Type-B_ activation functions consist of two real-valued functions, \(G_{||}(\cdot)\) and \(G_{\angle}(\cdot)\), which are applied to the magnitude (modulus) and phase (angle, argument) of the input tensor, respectively, as

\[G(\mathbf{z})=G_{||}(|\mathbf{z}|)\odot\exp(jG_{\angle}(\angle\mathbf{z})). \tag{16}\]

Table 4 details the Type-B activation functions included in \(\mathtt{complextorch}\). Where \(G_{\angle}(\cdot)\) is omitted, the phase information is unchanged and the activation is effectively a masking function as in Section 2.6. Additionally, a generalized Type-B activation function is included allowing the user to adopt any set of \(G_{||}(\cdot)\) and \(G_{\angle}(\cdot)\) desired.

#### 2.8.3 Fully Complex Activation Layers

Fully complex activation layers employ activation functions specifically designed for complex-valued data and hence do not have a general form. Table 5 details the fully-complex activation functions included in \(\mathtt{complextorch}\).

#### 2.8.4 ReLU-Based Complex-Valued Activation Layers

The ReLU is the most popular activation function in modern deep learning, and it has garnered significant attention in its extension to the complex domain. Most take a similar form to Type-A activation functions as in Section 2.8.1, operating on the real and imaginary parts of the input signal. However, some functions, like the Type-B \(\mathtt{modReLU}\) and fully-complex \(\mathtt{Guberman ReLU}\) (\(\mathtt{zReLU}\)) apply ReLU-like operations. Some efforts have been made to develop insights into how ReLU-based complex-valued activation functions "activate" across different regions and quadrants of the complex plane [35, 32]. Table 6 details the fully-complex activation functions included in \(\mathtt{complextorch}\), where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\).

### Complex-Valued Loss Functions

In this section, we overview some common complex-valued loss functions. Whereas we emphasize regression loss, complex-valued classification and other loss functions are further explored in [5, 2]. Similar to activation functions, two general types of loss functions have similar forms to Type-A and Type-B activations, operating on the real and imaginary or magnitude and phase, respectively.

\begin{table}
\begin{tabular}{c|c|c|c} \(\mathtt{complextorch}\) Activation Layer & \(G_{||}(|\mathbf{z}|)\) & \(G_{\angle}(\angle\mathbf{z})\) & Reference \\ \hline \(\mathtt{CVPolarTanh}\) & \(\tanh(|\mathbf{z}|)\) & - & Eq. (8) [27] \\ \hline \(\mathtt{CVPolarSquash}\) & \(\frac{|\mathbf{z}|^{2}}{(1+|\mathbf{z}|^{2})}\) & - & Section III-C [27] \\ \hline \(\mathtt{CVPolarLog}\) & \(\ln(|\mathbf{z}|+1)\) & - & Section III-C [29] \\ \hline \(\mathtt{modReLU}\) & \(\mathtt{ReLU}(|\mathbf{z}|+b)\) & - & Eq. (8) [30] \\ \hline \end{tabular}
\end{table}
Table 4: Type-B activation functions.

\begin{table}
\begin{tabular}{c|c|c} \(\mathtt{complextorch}\) Activation Layer & \(G_{\mathbb{R}}(\mathbf{z})=G_{\mathbb{I}}(\mathbf{z})\) & Reference \\ \hline \(\mathtt{CVSplitTanh}\) & \(\tanh(\mathbf{z})\) & Eq. (15) [27] \\ \hline \(\mathtt{CTanh}\) & \(\tanh(\mathbf{z})\) & Eq. (15) [27] \\ \hline \(\mathtt{CVSplitSigmoid}\) & \(\sigma(\mathbf{z})\) & - \\ \hline \(\mathtt{CSigmoid}\) & \(\sigma(\mathbf{z})\) & - \\ \hline \(\mathtt{CVSplitAbs}\) & \(|\mathbf{z}|\) & Section III-C [28] \\ \hline \end{tabular}
\end{table}
Table 3: Type-A activation functions.

\begin{table}
\begin{tabular}{c|c|c} \(\mathtt{complextorch}\) Activation Layer & \(G(\mathbf{z})\) & Reference \\ \hline \(\mathtt{CVSigmoid}\) & \(\frac{1}{1+\exp(\mathbf{z})}\) & Eq. (71) [31] \\ \hline \(\mathtt{zReLU}\) & \(\begin{cases}\mathbf{z}\ \
#### 2.9.1 Split Loss Functions

Split loss functions apply two real-valued loss functions to the real and imaginary parts of the estimated (\(\mathbf{x}\))) and ground truth (\(\mathbf{y}\))) labels as

\[\mathcal{L}(\mathbf{x},\mathbf{y})=\mathcal{L}_{\mathbb{R}}(\mathbf{x}_{ \mathbb{R}},\mathbf{y}_{\mathbb{R}})+\mathcal{L}_{\mathbb{I}}(\mathbf{x}_{ \mathbb{I}},\mathbf{y}_{\mathbb{I}}), \tag{17}\]

where the total loss is computed as the sum of the real and imaginary losses. Generally \(\mathcal{L}_{\mathbb{R}}(\cdot,\cdot)=\mathcal{L}_{\mathbb{I}}(\cdot,\cdot)\); however, we include an generalized split loss function allowing the user to specify any combination of \(\mathcal{L}_{\mathbb{R}}(\cdot,\cdot)\) and \(\mathcal{L}_{\mathbb{I}}(\cdot,\cdot)\). Table 7 details the split loss functions included in complextorch.

#### 2.9.2 Polar Loss Functions

Similarly, polar loss functions apply two real-valued loss functions to the magnitude and phase of the estimated and ground truth labels as

\[G(\mathbf{x},\mathbf{y})=w_{||}G_{||}(|\mathbf{x}|,|\mathbf{y}|)+w_{\angle}G_{ \angle}(\angle\mathbf{x},\angle\mathbf{y}), \tag{18}\]

where \(w_{||}\) and \(w_{\angle}\) are scalar weights applied based on _a priori_ understanding of the problem to scale the magnitude and phase losses, particularly as the phase loss will always be less than \(2\pi\) by definition. Tuning the loss weights may improve modeling performances To the author's understanding, at the time of this paper, there have been no efforts to apply polar loss functions. However, they may, in conjunction with split loss functions, improve modeling performance by imposing additional loss on the phase-accuracy of the algorithm.

#### 2.9.3 Other Complex-Valued Loss Functions

Several complex-valued loss functions are implemented and detailed in Table 8 Additionally, PerpLossSSIM (Eq. (5),

Fig. 1[38]) is implemented. Its mathematical formulation is omitted here, but can be found in [38].

### Complex-Valued Manifold-Based Layers

In [16, 18] a complex-valued convolution operator offering similar equivariance properties to the spatial equivariance of the traditional real-valued convolution operator is introduced. By approaching the complex domain as a Riemannian homogeneous space consisting of the product of planar rotation and non-zero scaling, they define a convolution operator equivariant to phase shift and amplitude scaling. Although their paper shows promising results in reducing the number of parameters of a complex-valued network for several problems, their work has not gained mainstream support. However, some initial work has shown significant promise in reducing model sizes and improving modeling capacity

\begin{table}
\begin{tabular}{c|c|c} complextorch Activation Layer & \(G(\mathbf{z})\) & Reference \\ \hline CVSplitReLU & \(\text{ReLU}(\mathbf{x})+j\text{ReLU}(\mathbf{y})\) & Eq. (5) [36] \\ \hline CReLU & \(\text{ReLU}(\mathbf{x})+j\text{ReLU}(\mathbf{y})\) & Eq. (5) [36] \\ \hline CPReLU & \(\text{PReLU}(\mathbf{x})+j\text{PReLU}(\mathbf{y})\) & Eq. (2) [35] \\ \hline \end{tabular}
\end{table}
Table 6: ReLU-based activation functions.

\begin{table}
\begin{tabular}{c|c} complextorch Loss Function & \(\mathcal{L}_{\mathbb{R}}(\cdot,\cdot)=\mathcal{L}_{\mathbb{I}}(\cdot,\cdot)\) \\ \hline SplitL1 & \(\text{L1}(\cdot,\cdot)\) \\ \hline SplitMSE & \(\text{MSE}(\cdot,\cdot)\) \\ \hline SplitSSIM & \(\text{SSIM}(\cdot,\cdot)\) \\ \hline \end{tabular}
\end{table}
Table 7: Split Activation Functions.

\begin{table}
\begin{tabular}{c|c|c} complextorch Activation Layer & \(G\mathcal{L}(\mathbf{x},\mathbf{y})\) & Reference \\ \hline CVQuadError & \(\frac{1}{5}\text{sum}(|\mathbf{x}-\mathbf{y}|^{2})\) & Eq. (11) [37] \\ \hline CVFourthPowError & \(\frac{1}{5}\text{sum}(|\mathbf{x}-\mathbf{y}|^{4})\) & Eq. (12) [37] \\ \hline CVCauchyError & \(\frac{1}{2}\text{sum}(c^{2}/2\ln(1+|\mathbf{x}-\mathbf{y}|^{2}/c^{2}))\) & Eq. (13) [37] \\ \hline CVLogCoshError & sum\((\ln(\cosh(|\mathbf{x}-\mathbf{y}|^{2}))\) & Eq. (14) [37] \\ \hline CVLogError & sum\((|\ln(\mathbf{x})-\ln(\mathbf{y})|^{2})\) & Eq. (10) [3] \\ \hline \end{tabular}
\end{table}
Table 8: Other complex-valued loss functions.
for smaller models [39, 40, 41]. Incorporating manifold-based complex-valued deep learning is a promising research area for future efforts. For full derivation and alternative implementations, please refer to [16, 17, 18].

As the authors mention in the final bullet point in Section IV-A1,

* "If \(d\) is the manifold distance in (2) for the Euclidean space that is also Riemannian, then wFM has exactly the weighted average as its closed-form solution. That is, our wFM convolution on the Euclidean manifold is reduced to the standard convolution, although with the additional convexity constraint on the weights."

Hence, the implementation closely follows the conventional convolution operator with the exception of the weight normalization. We would like to not that the weight normalization, although consistent with the authors' implementation, lacks adequate explanation from the literature and could be improved for further clarity.

complextorch contains implementations for 1-D and 2-D versions of the proposed wFM-based convolution operated introduced in [16, 18], dubbed wFMConv1d and wFMConv2d, respectively.

### Complex-Valued Pooling Layers

Complex-valued average pooling can be computed similarly to real-valued pooling where the average can be computed over the real and imaginary parts of the signal separately as

\[\texttt{CVAdaptiveAvgPoolingNd}(\mathbf{z})=\texttt{AdaptiveAvgPoolingNd}( \mathbf{x})+j\texttt{AdaptiveAvgPoolingNd}(\mathbf{y}), \tag{19}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\). We include implementations for \(\texttt{CVAdaptiveAvgPooling1d}\), \(\texttt{CVAdaptiveAvgPooling2d}\), and \(\texttt{CVAdaptiveAvgPooling3d}\).

### Complex-Valued Dropout Layers

Similarly, complex-valued dropout can be computed similarly to real-valued dropout where dropout can be computed over the real and imaginary parts of the signal separately as

\[\texttt{CVDDropout}(\mathbf{z})=\texttt{Dropout}(\mathbf{x})+j\texttt{Dropout }(\mathbf{y}), \tag{20}\]

where \(\mathbf{z}=\mathbf{x}+j\mathbf{y}\).

## 3 Conclusion

In this paper, we introduced a PyTorch wrapper for complex-valued neural network modeling. The proposed framework enables rapid development of deep learning models for signal processing and signal understanding tasks relying on complex-valued data. We detailed the implementation of deep learning layers spanning convolution, linear, activation, attention, and loss functions.
