A comparison of rational and neural network based approximations
Rational and neural network based approximations are efficient tools in
modern approximation. These approaches are able to produce accurate
approximations to nonsmooth and non-Lipschitz functions, including multivariate
domain functions. In this paper we compare the efficiency of function
approximation using rational approximation, neural network and their
combinations. It was found that rational approximation is superior to neural
network based approaches with the same number of decision variables. Our
numerical experiments demonstrate the efficiency of rational approximation,
even when the number of approximation parameters (that is, the dimension of the
corresponding optimisation problems) is small. Another important contribution
of this paper lies in the improvement of rational approximation algorithms.
Namely, the optimisation based algorithms for rational approximation can be
adjusted to in such a way that the conditioning number of the constraint
matrices are controlled. This simple adjustment enables us to work with high
dimension optimisation problems and improve the design of the neural network.
The main strength of neural networks is in their ability to handle models with
a large number of variables: complex models are decomposed in several simple
optimisation problems. Therefore the the large number of decision variables is
in the nature of neural networks.
Vinesha Peiris, Reinier Diaz Millan, Nadezda Sukhorukova, Julien Ugon
###### Abstract

###### Abstract

Rational and neural network based approximations are efficient tools in modern approximation. These approaches are able to produce accurate approximations to nonsmooth and non-Lipschitz functions, including multivariate domain functions. In this paper we compare the efficiency of function approximation using rational approximation, neural network and their combinations. It was found that rational approximation is superior to neural network based approaches with the same number of decision variables. Our numerical experiments demonstrate the efficiency of rational approximation, even when the number of approximation parameters (that is, the dimension of the corresponding optimisation problems) is small. Another important contribution of this paper lies in the improvement of rational approximation algorithms. Namely, the optimisation based algorithms for rational approximation can be adjusted to in such a way that the conditioning number of the constraint matrices are controlled. This simple adjustment enables us to work with high dimension optimisation problems and improve the design of the neural network. The main strength of neural networks is in their ability to handle models with a large number of variables: complex models are decomposed in several simple optimisation problems. Therefore the the large number of decision variables is in the nature of neural networks.

**A comparison of rational and neural network based approximations**

Vinesha Peiris, Reinier Diaz Millan, Nadezda Sukhorukova, Julien Ugon

**Mathematics Subject Classification (2020)**Primary: 41A50, 41A20, 41A63, 65D10, 65D12, 65D15

**Keywords:** Chebyshev approximation, rational approximation, neural network approximation, multivariate approximation

## 1 Introduction

In this paper we compare two efficient function approximations tools: rational approximation and neural network based approximation. These two types of approximation are very different in their nature, but both of them are able to produce fast and accurate approximations. We also consider some combinations of these two approaches in order to improve the performance of the learning system.

The neural network we are using in this study has a specific structure and also known as deep learning. Deep learning is one of the key tools in the modern area of Artificial Intelligence. There are many practical applications for deep learning, including data analysis, signal and image processing and many others [11, 33]. Despite all these practical applications, deep learning
is just a specific type of function approximation, where the function to be approximated is only known by its values in a finite number of points (training data), while the approximation prototype (class of approximations) is a composition of affine mappings and the so called activation functions (special type of univariate function). A very comprehensive and thorough textbook on the modern view on deep learning can be found in [11].

Our second main approach is rational approximation. In this case, the approximations are simply the ratios of polynomial functions. The choice of rational functions is quite natural: they provide flexible approximation to a wide range of functions, including non-smooth and non-Lipschitz function [19, 30, 31]. This flexibility is even comparable with free-knot piecewise polynomial approximation [28]. At the same time, the corresponding optimisation problems are quasiconvex [7, 18, 26] and there are a number of efficient computational methods to tackle them [9, 20, 24] just to name a few.

Our numerical experiments demonstrate that rational approximation is superior to neural network based approaches. Moreover, we also demonstrate that the optimisation based approach can be used to control the conditioning number of the constraint matrices appearing in auxiliary problems. This additional adjustment of the optimisation problems allows us to increase the size of the problems that can be solved. Therefore, the introduction of modern optimisation techniques lead to the enhancement of the learning system.

The paper is organised as follows. In section 2 we provide the background of the approximation methods we use in this paper: neural network, rational approximation and their combinations. In section 3 we explain the approximation models we are using in this paper. Then in sections 4-6 we compare the models. Finally, in section 7 we draw the conclusions and highlight possible future research directions. Appendix A.1 contains supplementary results, including the results with only two nodes in the hidden layer. The approximations with only two nodes in the hidden layer are not accurate, but we include these results for consistency and comparison.

## 2 Preliminaries

### Deep learning

Deep learning is a powerful tool for data and function approximation, which can also be used for data analysis and data classification. The power of deep learning is in its structure: the subproblems that the system has to solve are very simple from the point of view of modern optimisation theory and therefore the system can handle problems with hundreds and even thousands of decision variables very efficiently.

The origin of deep learning is mathematical in its nature. Essentially, the
objective of deep learning is to solve an approximation problem: optimise the weights (parameters) of the network. These weights are the decision variables of certain optimisation problems, whose objective functions represent inaccuracy of approximation. Therefore, it is natural to approach this problem using modern optimisation tools [11, 12, 33].

Solid mathematical background of deep learning was established in [10, 14, 17, 29]. These works rely on the results of the celebrated Kolmogorov-Arnold Theorem [3, 15]. The Kolmogorov-Arnold representation theorem states that every multivariate continuous function can be represented as a composition of continuous univariate functions over the binary operation of addition. In general, there is no algorithm for constructing these composition functions. Instead of constructing this representation, modern deep learning techniques approximate this composition function by a composition of affine transformations and the so-called activation functions. The activation functions are univariate non-polynomial functions. Most commonly used activation functions are sigmoid functions and Rectified Linear Unit (ReLU) functions (monotonic piecewise linear function \(\varphi(x)=\max\{0,x\}\)).

Most deep learning algorithms rely on least squares based measure of inaccuracy (loss function). Since loss functions are measures of inaccuracy, the goal of optimisation is to optimise the weights by minimising the corresponding loss function. Therefore, least squares-based models are minimising a smooth quadratic functions, and therefore fast and simple optimisation techniques (for example, gradient descent) are applicable. In some cases, however, other loss functions are more efficient: uniform based, etc.

The goal of this paper is to compare the approximation results obtained by deep learning and other approximation techniques. In particular, we are looking at rational approximation due to its approximation power [28]. More details on rational approximation will be provided in the next section. On the other hand, in [34], the author demonstrate that rational functions are as powerful as neural networks with standard ReLU activation functions. This result encouraged many researcher in the deep learning community to combine neural networks, rational functions and rational approximation techniques together to enhance the performance of each other [6, 22, 27].

### Rational approximation

Rational approximation in Chebyshev (uniform) sense was a very popular research direction in the 1950s-70s [1, 5, 8, 21, 30, 31] and many others.

There are two main groups of methods to approach rational approximation. The first group [23] is dedicated to "nearly optimal" solutions. This approach (also known as AAA approach) is very efficient and therefore very popular, but it is "nearly optimal". The extension of AAA to multivariate cases is still open. Moreover, this method is only designed for unconstraint optimisation and, as we will see later in this paper, this will limit our abil
ity to work with ill-conditioned matrices. The second group of methods is based on modern optimisation techniques: the corresponding optimisation problems are quasiconvex and can be solved using a general quasiconvex optimisation methods.

There are many methods for rational approximation [4, 16, 25, 30] (just to name a few), but the implementation of most of them relies on solving linear programming problems. Therefore, additional linear constraints can be easily added to these implementations without making the problems complex. Currently, the most popular optimisation methods for rational approximation are the bisection method for quasiconvex functions and the differential correction method. The advantage of these two methods is that these methods can be easily extended to the case of non-monomial cases (generalised rational approximation) [8, 26] and to multivariate settings [2]. Overall, the differential correction method has a quadratic convergence [4], while the bisection method converges linearly [7]. Therefore, in our experiments with univariate rational approximation we use the differential correction method. At the same time, the bisection method is still a good choice due to its simplicity. Hence, we use the bisection method in our multivariate rational approximation experiments.

## 3 Computational models

We compare six different approaches for approximating a given continuous function. Four of them are based on neural network (NN) and the remaining two are purely rational approximation based approaches. The approaches are as follows:

1. NN with ReLU activation function,
2. NN with rational approximation to ReLU activation (we apply the differential correction method to approximate ReLU),
3. NN with rational activation function, the coefficients of the rational activation function are learnt from NN,
4. NN with rational activation, the coefficients of the rational activation function and the parameters of the network are learnt with split method, where the training process is done in three steps: between the input layer and the hidden layer and then between the hidden layer and the output layer, and finally, the coefficients of the rational activation,
5. Rational approximation (Differential correction), this is the direct rational approximation between the input and output layer,
6. Rational approximation (AAA), this is the direct "near-optimal" rational approximation between the input and output layer.

We use neural networks with three layers: input layer, hidden layer and output layer. We have the following options for the number of nodes in the hidden layer:

* a network with 2 nodes in the hidden layer (see Appendix A.1),
* a network with 10 nodes in the hidden layer.

In our experiments, we use different numbers of epochs: 50, 100 and 200. We record the training time per epoch for each network training procedure.

For NN methods, we the MSE-based loss function (MSE stands for Mean Squared Error and simply means "least squares") and uniform loss. For the MSE-based loss we use ADAM, while for the uniform loss we use ADAMAX (a version of ADAM specially designed for uniform loss). All the rational based approximation models are designed for the uniform loss, which is a stronger criterion than least squares and clearly more appropriate if the goal is to approximate a function.

In our experiments, we consider two different settings.

* The domain is a segment (univariate function). We approximate the nonsmooth function \[f(x)=\sqrt{|x-0.25|},\quad x\in[-1,1],\] by using the above approaches with different conditions (number of nodes in the hidden layer, number of epochs, etc.). The choice of the function \(f(x)\) is due to its difficulty for most approximation techniques: this function is nonsmooth and non-Lipschitz at \(x=0.25\).
* The domain is bi-variate. It is also possible to extend the results to higher dimensions, but all these extensions are out of the scope of this paper.

In the next section, we only present the important results of the numerical experiments. In particular, we report the results related to neural networks with 10 nodes in the hidden layer where the loss function is in the form of uniform norm. The Appendix A contains a very thorough discussion on the remaining results.

## 4 Results: Neural network-based approximation.

The findings in this section are from a neural network with 10 nodes in the hidden layer. The loss function is based on the uniform norm and we use ADAMAX as the optimiser.
### Neural Network with ReLU activation

We start our experiments with the standard ReLU activation function. Table 1 demonstrates the loss function value and computational time for different epochs.

One can see that the value of the loss function does not improve significantly when the number of epochs is increasing. Moreover, it appeared that the minimum reported value of the loss function may take place before the final epoch. For example, in the case of 200 epochs, the minimal value (0.088032) was obtained at epoch 187, while the reported value (after 200 epochs) is 0.138172. The last column of this table reports the running time per epoch and there is no significant difference between the computational time for each epoch.

Figure 1 shows the approximations computed by the network with ReLU activation. The accuracy of the approximations is significantly better, compared to the networks with fewer nodes in the hidden layer. It also appeared that the approximation around the "difficult point" \(x=0.25\) is better in the case of the uniform loss than it is in the case of MSE (Figures for MSE can be found in the Appendix A). Therefore, the uniform loss may be a better measure when it is essential to obtain accurate approximations around such points.

### Neural Network with rational approximation to ReLU activation

In this section, we present the results of numerical experiments, where the experiment settings are similar to those in Section 4.1, but the activation

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Epoch & Final loss & Minimum loss & Run time (per epoch) \\ \hline
50 & 0.130828 & & 2.52s \(\pm\) 79.3ms \\
45 & & 0.117747 & \\ \hline
100 & 0.139669 & & 2.51s \(\pm\) 96.8ms \\
97 & & 0.111249 & \\ \hline
200 & 0.138172 & & 2.49s \(\pm\) 59.2ms \\
187 & & 0.088032 & \\ \hline \end{tabular}
\end{table}
Table 1: Results: experiments of NN with ReLU activation

Figure 1: Approximation is computed by a neural network with ReLU activation, 50, 100 and 200 epochs.
function is the rational approximation to ReLU. This approximation was found by the differential correction method.

Our activation function is a rational function of degree \((3,2)\): the rational approximation is the ratio of two polynomials, the degree of the numerator is \(3\) and the degree of the denominator is \(2\).

The coefficients of the rational activation function come from the best rational \((3,2)\) approximation to the ReLU function. These coefficients are fixed throughout the whole training procedure. We do not learn the coefficients with the rest of the parameters of the network.

Table 2 shows the improvement in the accuracy compared to the standard ReLU activation in Section 4.1. This observation is especially prominent when the number of epochs is \(100\) or \(200\).

Figure 2 shows that the accuracy of the approximation is improving, even for the "difficult point", especially when the number of epochs is \(100\) or more.

Overall conclusion: the approximation results are more accurate when the rational approximation to ReLU is used as the activation function rather than ReLU itself. This observation is valid for both MSE and uniform loss. Results related to MSE can be found in the Appendix A

### Neural Network with rational activation

In this case, our activation function is a rational function of degree \((3,2)\). The coefficients of the rational activation function are now a part of the parameter set. We learn these coefficients as we learn other parameters during

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Epoch & Final loss & Minimum loss & Run time (per epoch) \\ \hline
50 & 0.175439 & & 2.82s \(\pm\) 90.5ms \\
46 & & 0.125264 & \\ \hline
100 & 0.105346 & & 2.77s \(\pm\) 79.5ms \\
99 & & 0.071646 & \\ \hline
200 & 0.084491 & & 2.83s \(\pm\) 382ms \\
195 & & 0.053663 & \\ \hline \end{tabular}
\end{table}
Table 2: Results: experiments of NN with rational approximation to ReLU
the training procedure. This type of network with rational activation function whose coefficients are learnable parameters are called 'rational neural networks' and more details can be found in [6]. The Python code for this section is also from [6].

Table 3 shows that when the number of nodes in the hidden layer is 10, the optimal loss function values are very close to the case when ReLU was approximated by the rational function. Figure 3 confirms these results.

The approximation appears to be accurate even around the "difficult point" when the number of epochs is 200 (similar to MSE).

### Neural Network with rational activation, learn with split training method

In this section, our activation function is a rational function of degree \((3,2)\). The coefficients are now a part of the parameter set. Similar to the network in Section 4.4, we learn these coefficients as we learn other parameters during the training procedure. The training process is done in three steps:

1. weights between the input and hidden layer,
2. weights between the hidden layer and the output;
3. rational coefficients.

This approach is based on a block coordinate-wise optimisation; therefore, even when it produces a sequence which decreases the functional values, the limit point is not guaranteed to be globally optimal.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Epoch & Final loss & Minimum loss & Run time (per epoch) \\ \hline
50 & 0.131586 & & 2.75s \(\pm\) 77.9ms \\
44 & & 0.114195 & \\ \hline
100 & 0.104754 & & 2.64s \(\pm\) 62.1ms \\
96 & & 0.097785 & \\ \hline
200 & 0.078658 & & 2.76s \(\pm\) 423ms \\
177 & & 0.055616 & \\ \hline \end{tabular}
\end{table}
Table 3: Results: experiments with rational activation function

Figure 3: Approximation is computed by the usual training process, epoch is 50, 100, 200.
Table 4 shows that the results are comparable for this method and for the cases of rational activation and rational approximation to the ReLU activation function (in terms of the optimal values of the loss function).

Figure 4 shows with the increase of the number of epochs the approximation is improving even around the "difficult point".

Overall conclusion: the approximation results are more accurate when the rational activation function is used with the split training method rather than the usual training approach.

## 5 Direct rational approximation approach: the differential correction and AAA method.

In this section, all the experiments correspond to uniform loss. The decision variables are the coefficients of the rational function. We compare two main methods for constructing rational approximations: the differential correction method (converges to a global optimal solution) and the AAA method (practical method, converges to "near optimal" solution).

### The differential correction method

#### 5.1.1 Rational approximation of degree (21,20), the differential correction method.

When one considers a neural network with three layers and just 10 nodes in the hidden layer and the rational activation function of degree \((3,2)\), the out

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Epoch & Final loss & Minimum loss & Run time (per epoch) \\ \hline
50 & 0.168947 & & 2.65s \(\pm\) 72.2ms \\
49 & & 0.113316 & \\ \hline
100 & 0.093841 & & 2.71s \(\pm\) 662ms \\
75 & & 0.087595 & \\ \hline
200 & 0.061735 & & 2.82s \(\pm\) 384ms \\
190 & & 0.053655 & \\ \hline \end{tabular}
\end{table}
Table 4: Results: experiments with split method

Figure 4: Approximation is computed by the split training process, 50, 100, 200 epochs.
put is a rational function of degree \((21,20)\). Therefore, we can approximate the function \(f(x)=\sqrt{|x-0.25|}\) by a rational function of degree \((21,20)\) using the differential correction method and compare this approximation with the results obtained for uniform loss function and \(10\) nodes in the hidden layer. Results related to the case where the number of nodes on the hidden layer is \(2\), can be found in the Appendix A.

In theory, the direct approximation by a rational function of degree \((21,20)\) is more flexible. We use Python in our experiments.

We also compute the rational approximations of degree \((20,20)\) for comparison. We will also use these additional results to compare with the AAA method, which is expecting the same degree in the numerator and the denominator. Table 25 summarises the computational time and the optimal loss. The results demonstrate that the computational time is very small for all three settings and corresponding errors are very similar. Interestingly, the increase in the degree does not always improve the optimal loss function value. This can be explained as a result of computational instability when the dimension of the corresponding optimisation problems is increasing.

Table 5 summarises the computational time and the optimal loss values for the degrees (21,20), (20,20) and (21,21). These results are very similar.

Figure 5 depicts the approximation of degree (21,20). The approximation is accurate, but there is a strong oscillation of the approximation, which is not surprising for high degree rational functions.

Figure 5: Approximation by the differential correction method, rational approximation degree is \((21,20)\), \((20,20)\) and \((21,21)\) respectively.

\begin{table}
\begin{tabular}{|c|c|c|} \hline Degrees & Error (uniform norm) & Run time \\ \hline (21,20) & 0.04376536257200335 & 10.98 \\ \hline (20,20) & 0.04841949102524923 & 11.58 \\ \hline (21,21) & 0.04411128396473672 & 12.48 \\ \hline \end{tabular}
\end{table}
Table 5: Direct rational approximation for degrees \((21,20)\), \((20,20)\) and \((21,21)\) by the differential correction method.
### The AAA method

#### 5.2.1 Rational approximation of degree (21,20) with the AAA method

In order to compare the differential correction method with AAA, we apply the AAA method in the same experimental settings as they are in the case of the differential correction method. Due to the limitations of the AAA method, we cannot use different degrees for the numerator and the denominator. Hence, for these experiments, we use the rational approximation of degree (20,20) and the rational approximation of degree (21,21). The codes are developed in Python by C. Hofreither [13].

Table 6 and Figures 6 shows that the approximations are very accurate and the computational time is very low. These results suggest that the AAA method is very fast and accurate, even when the degree is high. The function and the approximation are indistinguishable from the picture.
\(u_{i}\) on \((x_{i},t_{i})\) by using the multivariate bisection method and the algorithm was implemented in Matlab. The specifications of the dataset is as follows:

* observation data (known function values) of size \(512\times 201\).
* spatio temporal points (known domain values).
* \(x_{i}\in[0,40]\) with the step size of \(1\).
* \(t_{i}\in[-20,19.84375]\) with the step size of \(0.390625\).

In [6], authors used Euclidean norm to compute the errors but here, we compute the error of the approximation in the uniform norm. Therefore, there are not many possibilities for comparison with the findings in [6]. All of our experiments are constructed by using a certain set of domain points (using every \(k^{\text{th}}\) point of each dimension and the corresponding function values). The approximation and the uniform error term are computed on the same reduced domain. This extra step has been taken to ensure that big data volume does not lead MATLAB programs to crash.

### The original dataset

The following pictures (Figure 7, Figure 8, Figure 9) are of the original dataset. Some images are constructed by using a certain set of domain points. In particular, we use the pairs of every \(20^{\text{th}}\) point of the domain (along each of the two dimensions) and pairs of every \(10^{\text{th}}\) point of the domain in order to reduce the dimension of the original problem. For simplicity, we sometimes refer to these pairs as "Every \(10^{\text{th}}\) point of the domain" or "Every \(20^{\text{th}}\) point of the domain".
Now, we compute rational approximations of different degrees and report the uniform approximation error and the computational time for each degree when \(k=20\) and \(k=10\). We start our experiments with rational approximation of degree \((2,2)\) and keep increasing the degree up to \((20,20)\). The classical monomials in the rational function were replaced by Chebyshev monomials for better approximations.

### Rational approximation of degree (2,2)

We report 3D and 2D pictures of the rational approximation of degree (2,2) along with its error curve.

Figure 8: Pictures constructed for the original dataset taking the pairs of every \(10^{\text{th}}\) point of the domain: 3D and 2D view

Figure 9: Pictures constructed for the original dataset taking all the points of the domain: 3D and 2D view
In Figure 10, we compute the approximation by only considering the pairs of every \(20^{\text{th}}\) point of the domain and in Figure 11, the approximation is computed on the domain which contains pairs of every \(10^{\text{th}}\) point of the domain. The computational time and the uniform error is presented in Table 7.

Figure 11: Approximations (above) and the error curves (below) are constructed on the domain which consists of pairs of every \(10^{\text{th}}\) point of the original domain: 3D and 2D view

Figure 10: Approximations (above) and the error curves (below) are constructed on the domain which consists of pairs of every \(20\)th point of the original domain: 3D and 2D view
One can clearly see that the algorithm runs faster when the number of discretised points in the domain is smaller.

### Rational approximation of degree (5,5)

Here, we now increase the degree of the approximation to degree (5,5).

\begin{table}
\begin{tabular}{|c|c|c|} \cline{2-3} \multicolumn{1}{c|}{} & Uniform error & Time (sec.) \\ \hline Every 20\({}^{\text{th}}\) point of the domain & 1.29449744612994 & 2.049840 \\ \hline Every 10\({}^{\text{th}}\) point of the domain & 1.36991465992617 & 2.504411 \\ \hline \end{tabular}
\end{table}
Table 7: Uniform error and computational time for the degree (2,2) approximation

Figure 12: Approximations (above) and the error curves (below) are constructed on the domain which consists of pairs of every 20\({}^{\text{th}}\) point of the original domain: 3D and 2D view.
Figure 12 and Figure 13 consist of the approximations computed on the domains which contain pairs of every \(20^{\text{th}}\) point of the domain and the \(10^{\text{th}}\) point of the domain, respectively. The computational time and the uniform error is presented in Table 8. The approximations are better than the degree (2,2) case and the uniform error is drastically reduced.

### Rational approximation of degree (10,10)

In this section, we approximate the original function by a rational approximation of degree (10,10). The approximations and the error curves are depicted in Figure 14 and Figure 15. Approximations resemble the original functions well in both domains.
The computational time and the uniform error are presented in Table 9. The uniform error term is much smaller than the case where the degree of the approximation is (5,5). On the other hand, the algorithm takes more time to compute the approximations as the degree increases,

Figure 14: Approximations (above) and the error curves (below) are constructed on the domain which consists of pairs of every \(20^{\text{th}}\) point of the original domain: 3D and 2D view.

Figure 15: Approximations (above) and the error curves (below) are constructed on the domain which consists of pairs of every \(10^{\text{th}}\) point of the original domain.
### Rational approximation of degree (18,18)

We now increase the degree of the approximation to (18,18). The approximations presented in Figure 16 and Figure 17 are very similar to the corresponding original functions presented in Figure 7 and Figure 8. One can also see from table 10 that the uniform error is much smaller than the previous cases.
### Rational approximation of degree (20,20)

In this section, we consider approximating the original functions in Figure 7 and Figure 8 by a rational function of degree (20,20). Even though the degree of the rational function is higher, the number of parameters (the coefficients of the polynomials in the numerator and the denominator) is smaller (\(\approx 400\)) compared to the number of parameters in the neural networks that the authors of [6] used in their experiments (\(\approx 8035\)).

As the degree of the rational function increases, the condition number of the constraint matrices appearing in the auxiliary problem of the bisection method increases and in some cases, the MATLAB code tends to complain and crash. This issue can be solved by adding an extra linear constraint to the auxiliary problem of the bisection method which restricts the denominator polynomial from above. The bisection method has the flexibility of adding constraints and this additional adjustment allows us to compute higher degree rational approximations. Similar observation can be found in [32] where authors use this property of the bisection method in matrix function evaluations. The experiments in this section where the degree of the approximation is \((20,20)\) were conducted with an additional constraint

\begin{table}
\begin{tabular}{|c|c|c|} \cline{2-3} \multicolumn{1}{c|}{} & Uniform error & Time (sec.) \\ \hline Every \(20^{\text{th}}\) point of the domain & 0.014061551772703 & 66.460222 \\ \hline Every \(10^{\text{th}}\) point of the domain & 0.034389125232539 & 167.995076 \\ \hline \end{tabular}
\end{table}
Table 10: Uniform error and computational time for the degree (18,18) approximation

Figure 17: Approximations (above) and the error curves (below) are constructed on the domain which consists of pairs of every \(10^{\text{th}}\) point of the original domain: 3D and 2D view.
which restricts the denominator from above.
Clearly, the approximations in Figure 18 and Figure 19 coincide with the original functions and the uniform error is much smaller compared to the lower degree approximations.

Overall conclusion: Our results cannot be compared with the results in [6] since the errors are computed in different norms. However, we demonstrate that the rational approximation in multivariate domain is as powerful as neural networks and it is capable of approximating complicated functions by using much fewer parameters than a neural network.

For each degree, we also computed the uniform error on the whole domain once the approximation is computed on a selected reduced domain. The results can be found in Table 12. Note that the in the case of degree (20,20), the denominator is bounded from above.

One can clearly see that the uniform error computed on the whole domain is very similar to the uniform error computed on the domain with fewer selected points (reduced domain) when the degree of the approximation is small. This observation is very prominent in the case where the domain consists of pairs of every \(10^{\text{th}}\) point of the domain. For higher degrees, the difference between the errors computed on the whole domain and reduced domain gets much larger when the domain has much fewer points (when the domain has pairs of every \(10^{\text{th}}\) point of the whole domain).

## 7 Conclusions and future research directions

In this paper we compared several approaches to approximating functions, each being a slight modification of the other one. Starting from a classical neural network with a ReLU activation function, we gradually changed our set up, first by replacing the activation function with a rational function approximating ReLU, then including the coefficients of the rational activation function in the learning parameters, and finally replacing the neural network

\begin{table}
\begin{tabular}{|c|c|c|} \cline{2-4} \multicolumn{1}{c|}{} & Degree & Uniform error \\ \hline \multirow{3}{*}{Pairs of every 20\({}^{\text{th}}\) point of the domain} & (2,2) & 1.49062088756304 \\ \cline{2-4}  & (5,5) & 1.57543434735584 \\ \cline{2-4}  & (10,10) & 245.34679368905 \\ \cline{2-4}  & (18,18) & 11404.1284016 \\ \cline{2-4}  & (20,20) & 88023.9109319522 \\ \hline \multirow{3}{*}{Pairs of every 10\({}^{\text{th}}\) point of the domain} & (2,2) & 1.40803015961408 \\ \cline{2-4}  & (5,5) & 0.702071289628257 \\ \cline{1-1} \cline{2-4}  & (10,10) & 0.334154891477345 \\ \cline{1-1} \cline{2-4}  & (18,18) & 15.3976686595557 \\ \cline{1-1} \cline{2-4}  & (20,20) & 9.191409640164363 \\ \hline \end{tabular}
\end{table}
Table 12: Uniform error computed on the whole domain
with a best rational approximation of the same number of parameters using traditional optimisation approaches.

Our finding is that as each step increases the flexibility and size of the feasible space, we observe an improvement in the approximation. In particular the optimisation-based approximation achieves better approximation than the neural network parts.

The training time is also in favour of the optimisation-based approximation, especially because it is often possible to achieve a good approximation with fewer parameters. As the dimension of the function to approximate increases, it is not clear whether this advantage remains.

Our study did not consider other aspects of the approximation, such as generalisability. It is well known in general that deep learning has good generalisability properties and avoid overfitting. However, it is not clear whether adding extra flexibility (for example including the coefficients of the activation function in the learning parameters) retain those generalisability properties. Further work needs to be done to gain better insights about this.

## Acknowledgements, Statements and Declarations

We are grateful to the Australian Research Council for supporting this work via Discovery Project DP180100602.
