Understanding Forward Process of Convolutional Neural Network
This paper reveal the selective rotation in the CNNs' forward processing. It
elucidates the activation function as a discerning mechanism that unifies and
quantizes the rotational aspects of the input data. Experiments show how this
defined methodology reflects the progress network distinguish inputs based on
statistical indicators, which can be comprehended or analyzed by applying
structured mathematical tools. Our findings also unveil the consistency between
artificial neural networks and the human brain in their data processing
pattern.
Peixin Tian
# Understanding the Forward Process of Convolutional Neural Networks.

###### Abstract

Despite widespread adoption and application in various domains, the interpretability remains a major challenge since the inception of this field. Understanding the decision-making process of these models, the dynamic changes occurring within the model as data is processed, and the appropriate adjustments of model parameters to cater to specific requirements can be quite challenging. Consequently, the lack of confidence in deep learning models engenders a diminished level of trust, impeding both their advancement in research and practical implementation. This research paper delves into a comprehensive analysis of the rotational transformations observed in the forward process of convolutional neural networks (CNN). It elucidates the activation function as a discerning mechanism that unifies and quantizes the rotational aspects of the input data. Experiments show how this defined methodology reflects the progress network distinguish inputs based on statistical indicators, which can be comprehended or analyzed by applying structured mathematical tools. Our findings also unveil the consistency between artificial neural networks and the human brain in their data processing pattern. Since the emergence of ChatGPT, its ability to exhibit some autonomous thinking and anti-human tendencies has increased people's concerns about the use of this technology. These discoveries empower us to quantify and intervene in the decision-making process of neural networks through the utilization of mathematical tools, thus bolstering the dependability and trustworthiness of artificial intelligence in applications.

**Keywords:** Explainable artificial intelligence, Convolutional neural network
## 1 Introduction

Artificial intelligence has made remarkable strides in domains such as computer vision, natural language processing and autonomous systems. Yet the inner workings of most AI models remain obscure and incomprehensible to humans, hindering our ability to scrutinize and evaluate their reasoning and decisions. This raises concerns about the trustworthiness, accountability, and ethics of AI systems, particularly when they are deployed in critical domains that affect human lives, property, and security. Explainable artificial intelligence (XAI) is a research field that aims to ensure that AI systems can justify their decisions [1, 2]. However, the current explanations of models are inadequate, and understanding and influencing the network decision processes remain major hurdle.

XAI methods can be classified into local and global approaches. Local approaches examine how different attributes affect the model decision for a given input, while global approaches measure the importance of high-level concepts for the model prediction. Local Interpretable Model-Agnostic Explanations [3] (LIME) is a local approach that generates a new dataset by perturbing the prediction of a complex model for a specific input and fits a simple model to the new dataset to explain the complex model locally. Lundberg and Lee [4] introduced a novel approach called Shapley Additive Explanations (SHAP) for interpreting model predictions. This method determines the significance of each feature in making a specific prediction. The framework encompasses a unique set of additive feature importance measures and provides theoretical substantiation for the existence of a singular solution within this category, which encompasses a multitude of advantageous characteristics.Testing with Concept Activation Vectors [5] (TCAV) is a global method that defines high-level concepts (such as texture, color, shape), and calculates their activation vectors at different levels in convolutional neural networks. Thus, obtaining a concept sensitivity score, which indicates the degree of dependence of the model on these concepts [1, 6, 7, 8, 9]. Besides above methods, XAI has witnessed numerous inspiring endeavors such as Grad-CAM [10], DeepLIFT [11] and Integrated Gradients [12]. Various strategies have been developed to address the issue of neural networks' lack of interpretability from a systematic standpoint [13, 14, 15, 16], yielding encouraging outcomes. In contrast to current XAI methods, this paper aims to understand and describe the data transformations in the network, rather than just understanding the influence of a certain attribute on the model prediction.

A powerful and versatile category of neural networks, known as convolutional neural networks, has been applied to a range of domains with great success. CNNs are composed of several fundamental elements: convolutional layers, pooling layers, activation layers, normalization layers, and fully connected layers. By integrating residual structures into CNNs [31], they have managed to surpass the inherent depth limitations of conventional networks, leading to improved performance. CNNs continue to be widely utilized due to their exceptional ability to generalize effectively and their remarkable computational efficiency, along with other advantages [17, 18].

Our goal was to elucidate the mechanisms of data transformation and prediction generation in a trained convolutional neural network across its various structures and levels. To accomplish this, we devise novel interpretations of some operations that
under preserved the input-output relationship. We designed experiments to demonstrate their validity. On this basis, we developed methods to assess and manipulate the model output using these interpretations.

Libby and Buschman [19] have revealed that the brain avoids interference in memory processing by selectively rotates sensory representations. This paper investigates how artificial neural networks process data by analyzing activation functions in terms of rotation. The representation reveals the data processing within the network and shows its similarity to information processing in the brain.

We acknowledge the limitations of our method in this paper. Our analysis is restricted to two-dimensional discrete convolution and the rectified linear unit [20, 21, 22] (ReLU) activation function, which are widely used in CNNs, but not universal. Therefore, our conclusion may not generalize to other types of CNNs.

## 2 Methods

Here we introduce a selection of methods and concepts that are employed in subsequent experiments, accompanied by comprehensive explanations.

### Two-dimensional discrete convolution

Two-dimensional discrete convolution is the most common convolution operation in convolutional neural networks when processing images. It can be expressed as below [23]:

\[H(i,j)=\sum_{m}\sum_{n}F(m,n)G(i-m,j-n) \tag{1}\]

After training, the model parameters are fixed, thus each convolution operation can be seen as a linear equation, and multiple-channel convolution filters form a system of linear equations. For instance, consider a 2 \(\times\) 2 convolution filter with 4 channels, \(G^{\prime}\), and input feature \(F^{\prime}\):

\[\begin{split} G^{\prime 1}=\left[\begin{array}{cc}k_{1}^{1}&k_{2}^{ 1}\\ k_{3}^{1}&k_{4}^{1}\end{array}\right],\quad G^{\prime 2}=\left[\begin{array}{cc}k_{ 1}^{2}&k_{2}^{2}\\ k_{1}^{2}&k_{1}^{2}\end{array}\right],\quad G^{\prime 3}=\left[\begin{array}{cc}k_{ 1}^{3}&k_{2}^{3}\\ k_{3}^{3}&k_{4}^{3}\end{array}\right],\quad G^{\prime 4}=\left[\begin{array}{cc}k_{ 1}^{4}&k_{ 2}^{4}\\ k_{1}^{4}&k_{1}^{4}\end{array}\right]\\ \end{split} \tag{2}\]

The corresponding linear equation system can be described as (ignore bias):

\[\begin{split}\begin{cases}F^{\prime}*{G^{\prime}}^{1}=k_{4}^{1} \cdot x_{1}+k_{3}^{1}\cdot x_{2}+k_{2}^{1}\cdot x_{3}+k_{1}^{1}\cdot x_{4}=y^ {1}\\ F^{\prime}*{G^{\prime}}^{2}=k_{4}^{2}\cdot x_{1}+k_{3}^{2}\cdot x_{2}+k_{2}^{ 2}\cdot x_{3}+k_{1}^{2}\cdot x_{4}=y^{2}\\ F^{\prime}*{G^{\prime}}^{2}=k_{4}^{3}\cdot x_{1}+k_{3}^{3}\cdot x_{2}+k_{2}^{ 3}\cdot x_{3}+k_{1}^{3}\cdot x_{4}=y^{3}\\ F^{\prime}*{G^{\prime}}^{2}=k_{4}^{4}\cdot x_{1}+k_{3}^{4}\cdot x_{2}+k_{2}^{ 4}\cdot x_{3}+k_{1}^{4}\cdot x_{4}=y^{4}\end{cases}\end{split} \tag{3}\]

Using matrix operations to express this process:
\[\left[\begin{array}{cccc}k_{4}^{1}&k_{3}^{1}&k_{2}^{1}&k_{1}^{1}\\ k_{4}^{2}&k_{3}^{2}&k_{2}^{2}&k_{1}^{2}\\ k_{4}^{3}&k_{3}^{3}&k_{2}^{3}&k_{1}^{3}\\ k_{4}^{4}&k_{3}^{4}&k_{2}^{4}&k_{1}^{4}\end{array}\right]\left[\begin{array}{ c}x_{1}\\ x_{2}\\ x_{3}\\ x_{4}\end{array}\right]=\left[\begin{array}{c}y^{1}\\ y^{2}\\ y^{3}\\ y^{4}\end{array}\right] \tag{4}\]

Matrix theory enables analyzing two-dimensional discrete convolution operations in the representation. In CNNs, channels commonly outnumber variables, and linear equations constrain the output differently. We measure the influence of each equation in output, select, and solve the most relevant ones, to describe data variation in the network.

### ReLU

ReLU [20, 21, 22] activation function dominates the current landscape of deep convolutional neural networks. Its appeal lies in its ability to accelerate convergence and promote sparsity in deep learning models. Its calculation is simple:

\[f(x)=max(0,x) \tag{5}\]

As in equation (5), the calculation of ReLU is simple. The activation function is mainly regarded as a kind of trigger mechanism. This discrete representation hampers analysis. Hence, this paper offers an alternative angle for analyzing ReLU. Here we show that ReLU performs a rotation of vectors in high-dimensional space. This selective rotation operation eludes a mathematical equation, but its effects on specific inputs can be assessed.

\[\mathbf{R}(\left[\begin{array}{c}x_{1}\\ x_{2}\\ \cdots\\ x_{n}\end{array}\right])=\left[\begin{array}{c}max(0,x_{1})\\ max(0,x_{2})\\ \cdots\\ max(0,x_{n})\end{array}\right] \tag{6}\]

\[\theta=\arccos\frac{\mathbf{v}\cdot\mathbf{v}^{\prime}}{\|\mathbf{v}\|\| \mathbf{v}^{\prime}\|} \tag{7}\]

As in equation (6), We define this selective rotation as \(\mathbf{R}(\mathbf{v})\), which shares the input and output with ReLU. \(\mathbf{R}(\mathbf{v})\) rotates a spatial vector by a minimum angle to align or orthogonalized it with the positive quadrant of the coordinate system defined by the basis vectors. Its scaling factor is \(\frac{k^{\prime}}{k}\), where \(k\) and \(k^{\prime}\) are the magnitudes of the original vector and the projection onto all positive subspaces, respectively. In this way, we can calculate the angle, \(\theta\), between high-dimensional space vectors and analyzing its mathematical properties before and after applying activation function as equation (7). We will demonstrate how defining such a rotation operation can reveal important insights in the following sections.

### Fully connected (FC) layer

We focus on the FC layers that are closest to the output here. Those that are inserted in the middle of the network for other purposes are beyond the scope of this paper.
The fully connected layer linearly transforms the data from the deep neural network and outputs the final distribution for classification tasks [24]. The layer parameters fit the network's data set distribution globally. The network assigns each data point to the most likely category channel and the confidence score reflects the similarity of this assignment.

Consider a network that classifies its inputs into \(n\) categories based on the output of its final hidden layer, which has c units. The FC layer with a \(c\times n\) weight matrix that maps the hidden output to the output distribution. To do this, the network first flattens the hidden output into a \(1\times c\) vector and then multiplies it by the weight matrix. The network then compares the resulting vector with each of the \(n\) category-specific vectors in the FC layer, which are fixed after training, and chooses the category with the highest similarity score as the output.

Projecting the data onto a low-dimensional subspace with maximum variance in the original space by Principal Component Analysis [25] (PCA) partly retains the global spatial information of the high-dimensional distribution that shows the structure of categories instead of their similarities or differences.

## 3 Results

To reveal the hidden shape of data, we first reduced its dimensions and assumed it formed a cone. Then we projected it onto different planes and looked for the sparsest clusters of points which marked the vertex of the cone. The principal axis was automatically calculated based on the vertex coordinates and the mean of the data. For clear visualization, we transform the data to a new coordinate system that has the cone's vertex as the origin and the principal axis as the z-axis.

We conducted dimensionality reduction to the parameters of fully connected (FC) layer in models trained on the ImageNet dataset [26]. As illustrated in figure 1, the parameters of the FC layers in these models do not follow a uniform distribution within the three-dimensional subspace. Instead, they exhibit a distinctive conical pattern. Interestingly, this pattern can be observed in previous studies exploring restricted Boltzmann machines [27], suggesting that it is not a result of the convolutional operations.

We utilized Python 3.8.2 for data analysis and visualization, and employed the libraries sklearn [28] (v.1.2.2), NumPy [29] (v.1.22.3) and Matplotlib [30] (v.3.5.2). The neural network was implemented in PyTorch (v.1.11.0) framework.
**Cone-shape distribution.** Figure 1 provides a comparative analysis between the distribution of FC layer parameters, which have been reduced to a three-dimensional subspace, and the standard conical surface. We utilized the ResNet [31] series networks, renowned for their widespread usage and excellent performance. ResNet exhibits a relatively straightforward architecture, comprising fundamental components necessary for the efficacy of deep neural networks, including convolutional layers, pooling layers, normalization layers, fully connected layers, and residual connections. We employed pre-trained ResNet models obtained from two prominent providers, PyTorch [32] and Microsoft [33]. Networks with matching depths from both providers exhibited comparable performance on the ImageNet dataset.

Beside ResNet, we present dimensionality reduction results for VGG19 [34] and Vision Transformer [35; 36], thereby showcasing the ubiquity of this phenomenon across diverse neural network architectures. The subsequent analysis will attempt to explain the emergence mechanism of the conical distribution.

To better evaluate the parameters after dimensionality reduction, we divided them into \(n\) equal parts along the z-axis direction. We calculate the area of the circumscribed circle of the convex polygon formed by the projection of the points in each part on the xy-plane. The area of circumscribed circle varies with the cross-section of distribution. How the area change curve approaches the standard quadratic curve reflects the similarity between the points and conical distribution. A sufficient number of points are needed to calibrate the cross-section of data distribution. Some slices, however, were too sparse to provide reliable information.

Figure 1: **Visualization of the parameters distribution.** We performed PCA on the FC layer parameters and normalized them to three dimensions. The yellow surface displays a standard conical distribution, with a principal axis indicated by the orange vector. We used open-source ResNet series models from **a,** Pytorch and **b,** Microsoft as well as **c,** VGG19 and **d,** VisionTransformer.
We performed experiments using models of different depths provided by Pytorch. According to the official documentation, Pytorch trained these models with relatively consistent parameters, thus minimizing the influence of specific training process. As shown in figure 2, the distribution after dimensionality reduction generally approached a conical with depth increasing. The aforementioned experiments do not serve as proof, but approximating the distribution as a cone can facilitate the analysis of rotations.

**Rotating cone.** When studying the cone-shaped distribution, a natural approach is to examine the distribution along the directions parallel and perpendicular to the principal axis. Through PCA, we projected the FC layers parameters onto the xz and xy planes [37], resulting in two-dimensional projected data distributions denoted as \(P_{xz}\) and \(P_{xy}\), respectively.

\(P_{xz}\) and \(P_{xy}\) are utilized to ascertain the relative positions of points corresponding to a specific category within the overall distribution and the cross-sectional projection

Figure 2: **Variation in the area of the parametric convex polygon’s circumcircle after dimensionality reduction.** The parameters of the dimensionality-reduced FC layer were partitioned into \(n\) segments. By plotting the circumcircle area of each segment’s convex polygon, we computed the deviation and p-value from the standard quadratic curve. We employed least-squares to evaluate the discrepancy between the data points and the curve. \(n=35;\text{ResNet34},P=2.8\times 10^{-4};\text{ResNet50},P=4.8\times 10^{-5}; \text{ResNet101},P=5.2\times 10^{-5};\text{ResNet152},P=5.5\times 10^{-7}.\) ***\(P<0.001\).
plane. We examine the vertical and polar coordinate of the same point under two scenarios: varying model providers with identical model architecture, and the same model provider with different model architectures. Subsequently, we perform statistical analysis to determine the relative positions of these points within the model distribution. In the case of \(P_{xz}\), the cone-shaped distribution exhibits non-uniformity along its principal axis. Consequently, our analysis focuses on the relative positions of points within the complete dataset, rather than their positions within uniformly distributed intervals. As for \(P_{xy}\), we transform the Cartesian coordinates into polar coordinates and rotate the coordinate system to align the first category precisely with the polar axis.

Fig. 3: **Inter-model similarity of FC layers.****a,**\(P_{xy}\) and \(P_{xz}\) are the results of projecting the data onto the xy-plane and xz-planes respectively. **b,** Convert \(P_{xy}\) in Cartesian coordinates to polar coordinates. **c,**\(P_{xz}\) and \(P_{xy}\) are halved by y-coordinate and first-class position. **d,** Calculate the proportion of categories with consistent partition to assess the similarity between inter-model FC layers. **e,** Comparison of projection \(P_{xy},P_{xz}\) similarity in the FC layer between PyTorch and Microsoft providers on the identical architecture model. **f,** Comparison of projection \(P_{xy},P_{xz}\) similarity in the FC layer provided by PyTorch on the identical architecture model.
As shown in figure 3 We partition the data within the entire dataset into two segments. The data from the same category tend to cluster together on the projection plane aligned with the principal axis, across models from various providers. On the other hand, the clustering probability decreases significantly on the projection plane perpendicular to the principal axis, indicating a more random distribution. Remarkably, models trained by PyTorch with its consistent training methodology show high consistency in both axial directions. These results suggest that neural networks perform a selective rotation operation that is related to the training process. This operation rotates data from different categories by varying angles in high-dimensional space, increasing the randomness in the projection planes orthogonal to the rotation axis.

Here we show that neural networks perform a selective rotation operation that is directly linked to the training process. This operation rotates data from different categories by varying angles in high-dimensional space, resulting in a higher degree of randomness in the projection planes perpendicular to the rotation axis. We illustrate this phenomenon using two scenarios of projecting high-dimensional data onto two-dimensional planes. In the first scenario, the distributions of \(p_{xz}\) and \(p^{\prime}_{xz}\) are more consistent, while the similarity between \(P_{xy}\) and \(P^{\prime}_{xy}\) is close to random. In contrast, in the second scenario, both directions show high consistency. Our findings reveal a novel aspect of neural network learning and suggest a possible way to improve their performance.

We define ReLU as a selective rotation operation as the equation (6). This operation poses certain challenges to capture with a unified mathematical equation, but we can quantify each ReLU operation based on specific inputs.

**Data rotation throughout the network.** In this paper, we analysis hidden layer features from two perspectives: feature maps and channels. Since the networks' output relies on the cross-channel information, inter-channel relationships can reflect the decision-making process better than feature maps. Different channels store distinct features extracted from the input, and that ReLU selectively rotates the hidden layer data. This rotation causes data from different channels to exhibit diverse characteristics. As a result, the cone-shaped distribution resulting from the rotation is observed among the data across channels.
We investigate the inter-channel parameters of the network's hidden layers using PCA. To achieve this, we flatten the feature maps within each channel into one-dimensional vectors and apply PCA to reduce the data dimensionality across all channels. We observe a cone-shaped distribution that emerges gradually with increasing layer depth (Figure 4). The data exhibit a uniform distribution before passing through the activation layer, indicating that this activation function causes the cone-shaped distribution.

We define ReLU as a rotation operation and show how this explains the network's data processing. The network transforms the input by channel-wise [38, 39] operations and feature-wise [40] down-sampling. According to equation (4), When the number of channels increases, convolution introduces conflicting information, thus reducing the number of channels can be advantageous to some extent. We analysis the high-dimensional vector angles among the dominant channels in the output of the nearest ReLU layer.

Figure 4: **Inter-channel dimensional reduction by PCA.****a-f,** The result of applying PCA to the input and to the output of the first convolutional layer, the first ReLU layer, the first, the second and the fourth residual layer. **g,** The overall structure of ResNet34. **A-F** represents the data position of **a-f**.
**Quantize rotation.** Figure 5**a** shows the rotated angle of data points for each category along the principal axis of the cone-shaped distribution, sorted from bottom to top. We show that ReLU processing leads to a substantial variation in the rotational probability of class-specific data points at different locations within the cone. This variation reflects the "density" of data points across the entire distribution, which is a key factor for identifying their positions. Because rotation introduces more uncertainty perpendicular to the principal axis, assessing data points based on their distribution-wide positions is more reasonable. Our experiment reveals a way to distinguish different classes along the principal axis.

Confidence levels also influence the rotation angles. We arrange the input in ascending order based on their confidence levels and measure the rotation angles by the dominant channels in the final layer.Figure 5**b** shows that the vector angles after ReLU processing the rotation angle grows, so does the corresponding increase in the level of confidence. This observation implies that the model assigns more confident classifications to inputs that undergo more substantial rotations.

Data rotation reveals intrinsic characteristics of data categories. We investigated how the angle of data rotation affects its position in the distribution and the network's confidence in its assessment. These findings suggest that data rotation can be used as a mathematical method to represent and manipulate the intrinsic characteristics of data categories.

It is challenging to track data rotation within the network due to the fluctuations in channel numbers and interlayer correspondence. The significant disparity in the count of unrotated classes highlights the network's varied input processing at different positions within the distribution. However, the experiment solely captures the data rotation dynamics in the later stage of the model.

**Role of rotation.** We also approach the analysis of feature maps from an alternative perspective. We consider each row of a feature map as an individual point in high-dimensional space, and collectively, the feature map represents a cluster of points.

**R

Fig. 5: **Relationship between rotation angle and input property.****a,** The changes of rotation angle along principal axis. **b,** The changes of rotation angle as the confidence level increases.
This thought is commonly used in video analysis, where each frame can be regarded as a fundamental unit. We extend this method to image analysis as well. By recognizing the correlation between adjacent rows in an image, this angle enables us to effectively locate distinct parts within the image.

We employed the t-Distributed Stochastic Neighbor Embedding [41] (t-SNE) technique for dimensionality reduction, aiming to preserve the intrinsic distance relationships among points in high-dimensional space. Figure 6 presents the t-SNE visualization results for the same input data across different layers of the model. Initially, the data points exhibit distinct clusters, which gradually merge and become indistinguishable as we move towards the final layers, reflecting the process of information integration within a single feature map. This process also reveals the occurrence of rotational phenomena. Unlike a simple linear one, this selective rotation operation varies depending on the input. It involves the gradual mixing and alignment of feature maps within a channel during the rotation and transformation process.

When classifying, the model needs to distinguish the main object from the background noise. We divide the data into two parts: the object and the noise. As the model rotates and transports features, the information within a single feature map converges and becomes indistinguishable. Figure 6 shows the dimensionality reduction results of the same input data in different layers of the model. The data points form clear clusters at first and blur at the end. We can exploit this characteristic to intervene in the model's decision-making in the early stages.

Figure 6: **Inner-feature dimensional reduction by t-SNE.****a-f,** The result of applying t-SNE to the input and to the output of the first convolutional layer, the first ReLU layer, the first,the second and the third residual layer. **g,** The overall structure of ResNet34. **A-F** represents the data position of **a-f**.
**Intervene making decision.** We utilized the K-means [42] clustering algorithm for binary classification of the dimensionally reduced data. We then set the two classes to zero in their respective corresponding channels or feature map rows and calculated the prediction. Figure 7**b** shows the decision intervention results of the same input at different depths of the hidden layers in the ResNet34 model. The feature-wise method outperformed the channel-wise method. Interestingly, the success rates of both methods displayed an inverted U-shaped trend with increasing network depth. This phenomenon reflects the data's mixing and separation dynamics during the selective rotation process in the network. As the rotation progresses, the data tends to become more homogeneous within the network initially, followed by a re-clustering

Fig. 7: **Comparing correction in channel-wise and feature-wise methods.****a,** Depicts how channel-wise and feature-wise approaches intervene a network’s misclassification of an image. The network’s confidence and output vary depending on the depth of intervention within the network. **b,** Shows the number of incorrect decisions that are corrected by intervening at different stages of the network and removing duplicate samples. From left to right, the outputs of the first convolutional layer, the first, second, third, and fourth residual layers, and the total are shown. The feature-wise and channel-wise methods are represented by blue and green bars, respectively, while the pink bar indicates the total count of misclassified samples by the model. The experiment involved 10,000 images.
phenomenon that often facilitates improved classification. By integrating suitable detection and decision algorithms, effective manipulation of data decisions within the network can yield favorable outcomes.

Figure 7**a** showcases the successful rectification of the model's inaccurate predictions through channel intervention preceding the initial residual layer. Despite not aligning with the ground truth labels, the succeeding layers' rectified outcomes generally correspond to the objects present in the images. Conversely, the feature map intervention method did not effectively address the mispredictions in this specific instances. As anticipated, our decision intervention approach based on clustering demonstrated superior performance during the initial stages of information fusion and rotation within the model.

## 4 Discussion

This study aims to elucidate the image data classification process carried out by mature networks and provides a mathematical reinterpretation of this process. We employed fundamental mathematical tools yet uncovered some valuable phenomena. Specifically, the rotational distribution and the understanding of the ReLU activation function exhibit remarkable similarities, offering a potential framework for a structured restatement approach. This discovery implies a significant correspondence between artificial neural networks and the human brain in terms of information processing, which is truly remarkable. Moreover, interpreting ReLU as a form of selective rotation opens new avenues for quantifying and influencing network decision-making processes. This paper offers preliminary insights into these possibilities and anticipates further advancements in future research to make practical strides in this field.

Due to constraints in space and capabilities, the vast potential of this theory remains unexplored. This paper focuses solely the ReLU activation function and does not offer a comprehensive explanation or proof for the generation process of the observed rotational distribution. Remarkably, the observed rotational distribution is not limited to networks utilizing the ReLU activation function. This suggests that the interpretive methodology employed in this study may have broader applicability to other activation functions, albeit with increased intricacy. Furthermore, this study does not track the rotational phenomenon of data throughout the network, fails to elucidate the precise meaning of the term "density" in categories, and does not provide practical quantification and intervention approaches. We eagerly anticipate subsequent research that can make more significant contributions, freeing artificial intelligence researchers and engineers from the black-box nature of artificial neural networks. Such work would unveil the relationship between artificial and human intelligence, propelling the field forward.
