Fractional Concepts in Neural Networks: Enhancing Activation and Loss
  Functions
The paper presents a method for using fractional concepts in a neural network
to modify the activation and loss functions. The methodology allows the neural
network to define and optimize its activation functions by determining the
fractional derivative order of the training process as an additional
hyperparameter. This will enable neurons in the network to adjust their
activation functions to match input data better and reduce output errors,
potentially improving the network's overall performance.
Zahra Alijani, Vojtech Molek
# Fractional Concepts in Neural Networks: Enhancing Activation and Loss Functions

###### Abstract

The paper presents a method for using fractional concepts in a neural network to modify the activation and loss functions. The methodology allows the neural network to define and optimize its activation functions by determining the fractional derivative order of the training process as an additional hyperparameter. This will enable neurons in the network to adjust their activation functions to match input data better and reduce output errors, potentially improving the network's overall performance.

## 1 Introduction

Fractional derivatives have been studied in the context of activation functions [21, 20], which can morph activation functions from one to another in a continuous manner. The activation function is a fundamental element within a neural

Figure 1: Fractional derivative of Mish and sigmoid activation functions.
network, introducing essential nonlinearity required to model complex relations in machine learning tasks, including classification and regression.

In this paper, we emphasize the importance of selecting appropriate activation and loss functions. Over the years, various neural network architectures have surfaced, each accompanied by its distinctive set of activation functions--ranging from the sigmoid, radial basis function (RBF), ReLU [5], Softplus [4], swish [16], Mish [14], and many more. However, the current practice predominantly relies on manual or heuristic selection of activation functions, often leading to an exhaustive trial-and-error methodology and frequent retraining of networks to find the optimal configuration.

Fractional calculus, on the other hand, opens doors to entire families of functions by naturally extending the original function through fractional derivatives. Here, the user only needs to specify the original function, while the fractional derivative is automatically adjusted. This elegant approach holds particularly true for activation functions. In contrast, for loss functions, the fractional derivative requires manual configuration.

However, the application of fractional activation functions comes with its unique set of challenges. One prominent challenge is the potential increase in computational complexity associated with fractional derivatives, which may impact the efficient implementation and execution of neural networks. Additionally, fractional activation functions demand careful tuning of specific hyperparameters, which can be intricate and time-consuming. Furthermore, the research on fractional activation functions remains relatively limited, leaving the practical effectiveness of these functions still an open question.

All of our code is publicly available1.

Footnote 1: [https://gitlab.com/irafm-ai/frac_calculus_01](https://gitlab.com/irafm-ai/frac_calculus_01)

## 2 Fractional Calculus

Fractional derivatives and integrals are used in neural networks to define a numerical trainable parameter value that automatically selects the optimal activation function for a particular task. This approach can be helpful when it is difficult to manually select the best activation function, as fractional derivatives and integrals allow for a more flexible and adaptive selection process. Furthermore, using fractional calculus can also improve the performance of neural networks by introducing more complex and nuanced mathematical models.

### Fractional Derivatives

Fractional calculus is a powerful mathematical tool to model various complex engineering and real-world systems. The three popular fundamental definitions of fractional derivatives are [7] :

* Grunwald-Letnikov [15] derivative of fractional order \(a\in\mathcal{R}^{+}\) : It is defined as \[D^{a}f(t)=\lim_{h\to 0}\frac{1}{h^{a}}\sum_{n=0}^{[\frac{t-n}{h}]}(-1)^{n}C_{n,a}f (t-nh),\] (1) where, \([x]\) is the integer part of \(x\) and \(C_{n,a}\) is the binomial coefficient.
* Riemann-Liouville derivative of fractional order \(a\in\mathcal{R}^{+}\): It is defined as \[D^{a}f(t)=\frac{1}{\Gamma(n-a)}\frac{d^{n}}{dt^{n}}\int_{a}^{t}\frac{f(\mu)}{( t-\mu)^{a-n+1}}d\mu,\] (2) for \(n-1<a<n,n\in\mathcal{Z}^{+}\) and \(\Gamma(.)\) is the Gamma function (we will recall it shortly).
* Caputo derivative of fractional order \(a\in\mathcal{R}^{+}\): It is defined as \[D^{a}f(t)=\frac{1}{\Gamma(n-a)}\int_{a}^{t}\frac{f^{n}(\mu)}{(t-\mu)^{a-n+1}}d\mu,\] (3) for \(n-1<a<n,n\in\mathcal{Z}^{+}\) where \(f^{n}(\mu)\) is the \(n\)-th order derivative of the function \(f(t)\).

It should be noted that for an initially relaxed function, all three definitions coincide. The Caputo definition, though more restrictive, is widely employed, as it allows the use of physical initial conditions.
The Gamma function, denoted by \(\Gamma(z)\), is a generalization of the factorial operator and is used to define the fractional derivative in fractional calculus. The Gamma function is defined as [1]:

\[\Gamma(z)=\int_{0}^{\infty}t^{(z-1)}e^{-t}dt. \tag{4}\]

The Gamma function is defined for non-negative integers as \(\Gamma(n)=(n-1)!\), and for other non-negative values of \(z\) it can be computed by [1]:

\[\Gamma(z)=\frac{e^{-\gamma z}}{z}\prod_{k=1}^{\infty}((1+\frac{z}{k})^{-1}e^{ \frac{k}{k}}). \tag{5}\]

Here, \(\gamma\) is the Euler-Mascheroni constant \((\gamma=0.57721..)\). The fractional derivative, represented above, can be modified by replacing the factorial with the gamma function. The definition provided in the following statement represents the fractional derivative of the function \(f(x)=x^{k}\) for \(k,x\geq 0\).

\[D^{a}x^{k}=\frac{\Gamma(k+1)}{\Gamma(k+1-a)}x^{k-a}. \tag{6}\]

The Gamma function allows for the definition of the fractional derivative for non-integer values of \(k\), while the factorial is only defined for integers. This allows for a more general and flexible formulation of the fractional derivative. In machine learning, fractional derivatives and integrals have been used in various ways. For example, they have been used to design activation functions, as discussed in [21]. They will also be used in the design of loss functions, where they can capture more complex patterns in data.

## 3 Grouping Activation Functions by Using Fractional Calculus

It is possible to group certain activation functions into families using fractional calculus. By analyzing the fractional derivative of an activation function, it is possible to mathematically generate other activation functions that belong to the same family.

It is worth noting that not all activation functions can be grouped into families using fractional calculus. The step function is an example of an activation function that is computationally efficient but is a poor choice for an activation function due to discontinuity and flat derivations. Other available activation functions, such as sigmoid, ReLU, and tanh, each with its own strengths and weaknesses, provide detailed information on the various options [19].

It also depends on the type of neural network and the problem you are trying to solve. It is also important to note that fractional calculus is a relatively new area of research and is still being fully understood. Its use in neural networks is still experimental and not widely used.

Calculating the fractional derivative of some activation functions is challenging and may not have a simple closed-form expression. However, there are approximation methods, i.e., the Grunwald-Letnikov [15] or other specialized techniques, that are generally used to calculate fractional derivatives. These methods involve approximating the fractional derivative using finite differences or numerical integration.

### Fractional Mish

Mish is mathematically defined as:

\[f(x)=x\tanh(\ln(1+e^{x})))=x.\frac{(e^{x}+1)^{2}-1}{(e^{x}+1)^{2}+1}.\]

Some traditional activation functions, such as sigmoid and tanh, suffer from the saturation problem, where the gradients become very small for significant inputs. The Mish activation (and its fractional version) helps mitigate this issue by having bounded angles for substantial positive and negative inputs, leading to more stable training. The Mish activation has been reported to perform well in particular machine learning tasks, such as image classification. The fractional Mish can be computed as:

\[D^{a}f(x)=\lim_{h\to 0}\frac{1}{h^{a}}\sum_{n=0}^{\infty}(-1)^{n}\frac{ \Gamma(a+1)(x-nh)}{\Gamma(n+1)\Gamma(1-n+a)}\cdot\frac{(e^{x-nh}+1)^{2}-1)}{( (e^{x-nh}+1)^{2}+1)}. \tag{7}\]

The Mish activation function combines the linear and non-linear properties of the identity and hyperbolic tangent functions, respectively. It has been shown to improve the training performance of deep neural networks in some cases.
### Fractional sigmoid

The sigmoid activation function takes an input value and maps it into a value between 0 and 1. One characteristic of the sigmoid function is its tendency to saturate as the input becomes large or small. The saturation behavior leads to the vanishing gradient problem, where the gradients approach zero during backpropagation, hindering the learning process. The fractional sigmoid activation function can be implemented using the soft plus function or directly by applying the fractional derivative to the sigmoid function. The sigmoid function is defined as:

\[f(x)=\frac{1}{1+e^{-x}}\]

and the fractional sigmoid is defined as:

\[D^{a}f(x)=\lim_{h\to 0}\frac{1}{h^{a}}\sum_{n=0}^{\infty}(-1)^{n}\frac{\Gamma(a+1)}{ \Gamma(n+1)\Gamma(1-n+a)(1+e^{-x+nh})} \tag{8}\]

where \(a\) is the fractional derivative order. By adjusting the value of \(a\), we can control the shape and smoothness of the activation function.

Sigmoid is an outdated activation function primarily used in the last layer of networks to squeeze an output into \([0,1]\) interval. Fractionality makes sigmoid usable and alleviates gradient vanishing, albeit it does not remove it completely.

## 4 Grouping Loss Functions by Using Fractional Calculus

The term "fractional" suggests that these loss functions may involve fractional exponents or weights, which can be used to fine-tune the contribution of individual data points or classes. Fractional loss functions can have a more complex form than traditional loss functions and may combine various components to achieve specific objectives. Here is an example of a loss function that utilizes fractional calculus in a time series prediction problem:

\[L(y,\hat{y})=\int_{t_{1}}^{t_{2}}w(t)D^{a}(y(t)-\hat{y}(t))dt\]

In this equation, \(y(t)\) represents the actual label or target value, \(\hat{y}(t)\) represents the predicted value by the model, \(a\) is the fractional derivative order, \(D^{a}\) represents the fractional derivative operator, and \(w(t)\) is a weight function that assigns weights to different time points or data points. The integral is taken over the interval \([t_{1},t_{2}]\), which means the specified time interval. It is important to note that the specific choice and formulation of the loss function depends on the problem at hand and the desired properties or characteristics that need to be incorporated. Despite having a trainable parameter in the calculation of the fractional activation function, for the loss function, \(a\) is a non-trainable parameter.

### Fractional Mean Squared Error

Machine learning aims to find a model that can accurately predict the output given the input data. This is typically done by minimizing a loss function, which measures the difference between expected and actual output. The model parameters are then adjusted to reduce the loss function. A standard loss function used in many machine learning algorithms is the mean squared error (MSE), which is defined as:

\[\text{MSE}(y,\hat{y})=\frac{1}{n}\sum(y-\hat{y})^{2}. \tag{9}\]

\(y\) is the proper output, \(\hat{y}\) is the predicted output, and n is the number of samples. The goal is to minimize the MSE by adjusting the model parameters. One example of a loss function with a fractional derivative is the fractional mean squared error (FMSE), which is defined as:

\[\text{FMSE}(y,\hat{y},a)=\frac{1}{n}\sum\big{(}(y-\hat{y})^{2}\big{)}^{\frac{ 9}{2}}, \tag{10}\]

where \(a\) is the order of the fractional derivative. When \(a=1\), the FMSE is equivalent to the MSE. However, when \(a\) is a non-integer value, the FMSE captures more complex relationships between the input and output.
### Fractional Cross-Entropy Loss

The fractional cross-entropy loss is a loss function with a fractional derivative that can be used in classification problems to improve the model's accuracy. The fractional cross-entropy loss function is defined as:

\[\text{FCrossEntropy}(y,\hat{y},a)=-\frac{1}{n}\sum(ya\log(\hat{y})+(1-y)a\log(1 -\hat{y})) \tag{11}\]

One potential limitation of using loss functions with fractional derivatives is that they may require more computational resources and may be more challenging to optimize than standard loss functions. Additionally, the choice of the order of the fractional derivative may not be evident in all cases and may require some trial and error.

### Fractional Huber Loss

The fractional Huber loss is a modified version of the Huber loss [8] that incorporates a fractional derivative to capture more complex relationships between variables. Huber loss is a robust function combining squared error for minor errors and absolute errors for significant ones. By adjusting the "a" value, the fractional Huber loss can adapt to different levels of non-linearity and robustness in the data. Additionally, the delta parameter allows control over the point at which the loss function transitions from quadratic to linear. The fractional Huber loss is particularly useful when the data contain outliers or extreme values, as it provides a robust alternative to the standard squared error loss. A fractional version of the Huber loss can be defined as:

\[\text{FHuberLoss}(y,\hat{y},a)=\left\{\begin{array}{ll}\frac{1}{2}((y-\hat{y })^{2})^{\frac{\delta}{2}},&\text{if}\quad|y-\hat{y}|\leq\delta,\\ \delta(|y-\hat{y}|)^{\frac{\delta}{2}}-\delta^{2}(\frac{1-a}{2}),&\text{ otherwise}.\end{array}\right. \tag{12}\]

Here \(n\) is the number of samples, a is the order of the fractional derivative, and \(\delta\) is a threshold determining the point at which the loss transitions from quadratic to linear.

## 5 Activation function results

In our experimental study, we evaluated the performance of three activation functions: Mish, swish, and sigmoid, on the CIFAR-10 dataset [10]. These evaluations were conducted across four distinct network architectures: a simple CNN, ResNet 18 and 50 [6], and an architecture resembling EfficientNetB0 [18]. For detailed insights into the architecture specifics, we encourage readers to refer to our repository. We should note that we utilized the fractional swish implementation in [20], but, despite our efforts, we could not surpass the performance of the original swish activation function.

Setup:We use the common CIFAR-10 setting. SGD optimizer with a five epochs warm-up and decreasing learning rate at 30%, 60%, and 80% milestones. Data are fed to the network in batches of 128 images augmented with padded random cropping, horizontal flipping, and normalization. The network is trained for 200 epochs using a cross-entropy with label smoothing. There is an exception: ResNet-50 fractional Mish is trained with 96 batch size due to memory consumption. We track the best test accuracy and the fractional derivative order of each activation function during training. All activation functions have hyperparameters \(h=1\) and \(\sum_{n=0}^{3}\) (see function definitions). Sec. 7 discusses these two constants in further detail.

Table 1 displays the CIFAR-10 test accuracies for various combinations of models and activation functions. The results for Fractional Mish and Swish are somewhat underwhelming, as they do not exhibit a signif

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Act fc** & \begin{tabular}{c} **Simple CNN** \\ [\%] \\ \end{tabular} & \begin{tabular}{c} **ResNet-18** \\ [\%] \\ \end{tabular} & \begin{tabular}{c} **ResNet-50** \\ [\%] \\ \end{tabular} & 
\begin{tabular}{c} **EffNet** \\ \end{tabular} \\ \hline ReLU & 75.97 & 95.38 & 95.42 & 94.40 \\ \hline Sig & **65.69** & 82.01 & 56.49 & 84.16 \\ Frac sig & 10.00 & **94.39** & **90.77** & **92.21** \\ \hline swish & **77.53** & **94.31** & **94.62** & **93.92** \\ Frac swish & 77.40 & 93.42 & 93.82 & 92.97 \\ \hline Mish & **77.67** & **94.22** & **94.90** & **93.55** \\ Frac Mish & 77.46 & 94.07 & 94.06 & 93.34 \\ \hline \hline \end{tabular}
\end{table}
Table 1: CIFAR-10 test accuracies of different model/activation function combinations.
the original activation functions. On the other hand, the Fractional Sigmoid outperforms the original function notably, except in the case of the simple CNN model, where the entire model fails to converge completely. It's noteworthy that Sigmoid performs significantly worse with ResNet-50 compared to both ResNet-18 and EfficientNetB0. To ensure that this behavior is not solely due to seeding issues, we conducted an additional run without seeding or deterministic algorithms, resulting in a final accuracy of 43.44%. One plausible explanation for this discrepancy is the vanishing gradient issue, particularly as network depth increases.

Figures 1(a) and 1(b) provide insights into the evolution of fractional derivative orders for Fractional Mish and Sigmoid. Notably, most fractional derivative orders for these activation functions tend to converge toward similar values. In the case of Fractional Sigmoid, convergence is observed toward its second derivative, whereas Fractional Mish remains closely aligned with the original function.

## 6 Loss function results

In this set of experiments, we explore the effectiveness of three distinct loss functions: fractional cross-entropy, fractional mean squared error (MSE), and fractional Huber loss. Fractional cross-entropy is evaluated on CIFAR-100LT [2] and ImageNet [17], while fractional MSE and fractional Huber loss are tested using the UTKFace regression dataset [22].

Figure 2: Evolution of fractional derivative order during the training of ResNet-18 on CIFAR-10.
Fractional cross-entropy and CIFAR-100LT:We use Adam optimizer with a decreasing learning rate at 60%, 80%, and 92.5% milestones. Data are fed to the network in batches of 50 images augmented with translation, horizontal flipping, and normalization. The network is trained for 200 epochs. Learning rates vary between experiments because fractionality adds a power term to the cross-entropy equation. In Fig. 2(a), we present a visual representation of the impact of fractional derivative order and learning rate on the final test accuracy. Lower learning rates tend to benefit from fractional derivative orders less than 1, whereas higher learning rates exhibit improved performance with fractional derivative orders greater than 1.

Fractional cross-entropy and ImageNet:We use Adam optimizer with a decreasing learning rate at 33% and 66% milestones. Data are fed to the network in batches of 256 images augmented with random resized cropping, horizontal flipping, and normalization. The network is trained for 90 epochs. As illustrated in Fig. 2(b), the model's accuracy with a fixed learning rate of 2e-3 exhibited minimal changes and peaked at 0.6.

Fractional Huber/MSE loss and UTKFace:We use the same experiment setting as CIFAR-10 in sec. 5 except for using Lion optimizer. We chose Lion over SGD due to its performance in this particular experiment. We run training with different fractional derivative orders. Table 2 summarizes the lowest losses achieved across different network architectures. It's important to note that, as the loss function itself is being tuned, we did not employ it for reporting test loss values. Instead, we utilized the L1 loss (Mean Absolute Error - MAE) to assess the test performance. The fractional loss function served the sole purpose of training during this experiment.

## 7 Practical challenges

As we transition from the theoretical framework of fractional calculus to practical implementation, we encounter several specific challenges, which we'll address in this section.

\begin{table}
\begin{tabular}{l|c c} \hline \hline
**Loss function** & **Simple CNN** & **ResNet-18** \\ \hline MSE & \(8.702\) & \(6.142\) \\ Fractional MSE & \(\mathbf{8.319}\) (1.6) & \(\mathbf{5.982}\) (0.9) \\ \hline Huber loss & \(8.678\) & \(6.046\) \\ Fractional Huber loss & \(\mathbf{8.174}\) (3.4) & \(\mathbf{6.042}\) (1) \\ \hline \hline \end{tabular}
\end{table}
Table 2: UTKFace test MAE of Baseline MSE and Huber loss and their fractional variants. The numbers in parentheses are fractional derivative orders.

Figure 3: Effect of the fractional derivative order of cross-entropy and learning rate on test accuracy.
Computational hyperparameters:One of the key challenges lies in setting appropriate computational hyperparameters, specifically the value of \(h\) and the number of sigma iterations. To illustrate this, let's begin with a fundamental derivative theorem:

\[L=\lim_{h\to 0}\frac{f(b+h)-f(b)}{h} \tag{13}\]

and its application to a sigmoid function. We must substitute \(lim_{h\to 0}\) with \(h\) equal to a scalar value. Fig. 4 shows the effect of increasing \(h\). With increasing \(h\), the derivative graph shifts to the right. This is true for both eq. 13 and eq. 8.

In Eq.8, we have the option to increase the number of sigma iterations to obtain a more precise approximation (see Fig.5). However, this approach introduces its own set of challenges. Increasing the number of iterations results in slower computation, potential issues related to type overflow due to the factorial function \(\Gamma(1+n)=n!\), the occurrence of negative integer arguments for the gamma function, and problems associated with memory consumption.

Modern deep learning frameworks, like PyTorch, construct a computational graph to track values relevant to gradient computations. In this context, each sigma iteration becomes a part of the computational graph, causing memory consumption to grow rapidly (see Table 3).

The problem of a negative, integral gamma function argument comes from the term \(\Gamma(1-n+a)\) when \(a\in\mathcal{N}^{+}\); therefore, it is impossible to increase the number of iterations of the sigma beyond a certain point. In general, \(n<1+a\) therefore, we impose the restriction that \(a\in[0.1,1.9]\) and \(\sum_{n=0}^{3}\). To maintain the constraint on \(a\) within the interval \([0.1,1.9]\), we employ a leaky piecewise-linear function:

\[f(x)=\begin{cases}0.1+(x-0.1)*0.01,&x<=0.1\\ x,&0.1<x<1.9\\ 1.9+(x-1.9)*0.01,&x>=1.9\end{cases}.\]

Figure 4: Left: first derivative of sigmoid using eq. 13. Right: first derivative of sigmoid using fractional derivative eq. 8. The first derivative of a sigmoid function with variable \(h\).

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Act fc** & \begin{tabular}{c} **Simple CNN** \\ [MiB] \\ \end{tabular} & \begin{tabular}{c} **ResNet-18** \\ [MiB] \\ \end{tabular} & \begin{tabular}{c} **ResNet-50** \\ [MiB] \\ \end{tabular} & 
\begin{tabular}{c} **EffNet** \\ [MiB] \\ \end{tabular} \\ \hline ReLU & 1 867 & 2 961 & 6 081 & 5 017 \\ \hline Sig & 1 867 & 2 963 & 6 075 & 5 165 \\ Frac sig & 1 897 & 5 623 & 20 503 & 15 775 \\ \hline swish & 1 867 & 3 203 & 7 533 & 6 045 \\ Frac swish & 1 893 & 4 597 & 14 685 & 11 585 \\ \hline Mish & 1 867 & 3 131 & 7 541 & 6 237 \\ Frac Mish & 1 955 & 10 517 & OOM & 26 059 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Memory consumption of different activation functions/models using CIFAR-10.
This formula enables the argument \(x\) to move outside the \([0.1,1.9]\) range when necessary. The slope or leakiness factor of 0.01 allows the fractional derivative order to recover from deviating outside these bounds.

All the problems mentioned above can be relaxed using different definitions for fractional derivatives, which is possible in particular cases. For example, fractional derivation of swish function eq. (28) in [20]: \(D^{a}g(x)=g(x)+a\sigma(x)(1-g(x))\).

## 8 Ablation study

Optimizers:One of the aspects we explored is the impact of optimizers on the training process of fractional derivative activation functions. These functions exhibit changes in shape during training, and it's unclear whether stochastic gradient descent (SGD) allows them to converge to the final fractional derivative order quickly enough. To test this convergence speed hypothesis, we experimented with various optimizers, including Lion [3], AdaBound [13], Adam [9], AdamW [12], and RAdam [11].

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**SGD** & **Lion** & **AdaBound** & **Adam** & **AdamW** & **RAdam** \\ \([\%]\) & [\%] & [\%] & [\%] & [\%] & [\%] \\ \hline
**94.45** & 92.01 & 91.15 & 92.52 & 92.72 & 92.19 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Test accuracy of ResNet-18 trained on CIFAR-10 using different optimizing algorithms.

Figure 5: Different number of sigma iteration and h values. Graphs compare the sigmoid’s first derivative (blue) and 1.1 derivative (green).
Table 4 displays the test accuracy of ResNet-18 trained on CIFAR-10 using different optimizing algorithms. We adhered to the optimal values recommended in the respective optimizer papers and for the experiments' general settings, as outlined in Section 5.

The activation functions in question are notably sensitive to the initial value of the fractional derivative order. To identify the optimal initial settings for both the fractional sigmoid and fractional Mish, we utilized 20% of the CIFAR-10 dataset and the Optuna framework for a search.

As shown in Fig.6, these results align with Fig.2. The fractional sigmoid tends to converge close to the second derivative, while the fractional Mish converges towards the original function.

Block activation functions:In the ResNet architecture, layers are organized into blocks. We investigated the impact of using a unique activation function with a fractional derivative order per layer compared to a shared activation function with a fractional order per block (meaning the same activation function shape is employed within a block). To find a difference, we used ResNet18 and trained it on CIFAR-10 with the following result:

As Table 5 demonstrates, the difference between employing a fractional derivative order per layer and per block is insignificant. Consequently, we have adopted the per block variant for all our experiments.

## 9 Conclusion

This paper introduces a novel technique for integrating fractional concepts into neural networks, enabling the modification of activation and loss functions. We leverage fractional derivatives to create families of activation and loss functions. In this framework, the fractional derivative order becomes a trainable parameter for activation functions,

Figure 6: CIFAR-10 test accuracy with respect to the initial fraction derivative of the factional Mish and sigmoid activation functions. Trained on 20% of data.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & **Per layer** & **Per block** \\  & [\%] & [\%] \\ \hline
**Frac Sig** & 94.29 & 94.39 \\
**Frac Mish** & 94.05 & 94.07 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Fractional derivative order per layer versus per block.
while it remains non-trainable for loss functions. The potential for shaping activation functions offers promise, particularly in situations where conventional functions, such as sigmoid, prove suboptimal.

However, we have encountered various challenges and obstacles throughout our exploration. The computation of fractional derivatives using the Grunwald-Letnikov method results in infinite-series representations, necessitating a trade-off between approximation accuracy and computational feasibility. Additionally, alternative calculation variants often involve multiple variables, making it challenging to integrate fractional derivatives into neural networks seamlessly.

The experimental results presented in our study underscore the potential benefits of fractional sigmoid and the fractional loss functions. Fractional sigmoid exhibits the ability to address some of the limitations encountered by the conventional sigmoid function. Moreover, the fractional loss functions consistently outperform their non-fractional counterparts, although they require the manual selection of fractional derivative orders.

## Acknowlegement

The work of Zahra Alijani was partially supported by project AIMet4AI No. CZ.02.1.01/0.0/0.0/17_049/0008414.
