Smooth Mathematical Function from Compact Neural Networks
This is paper for the smooth function approximation by neural networks (NN).
Mathematical or physical functions can be replaced by NN models through
regression. In this study, we get NNs that generate highly accurate and highly
smooth function, which only comprised of a few weight parameters, through
discussing a few topics about regression. First, we reinterpret inside of NNs
for regression; consequently, we propose a new activation function--integrated
sigmoid linear unit (ISLU). Then special charateristics of metadata for
regression, which is different from other data like image or sound, is
discussed for improving the performance of neural networks. Finally, the one of
a simple hierarchical NN that generate models substituting mathematical
function is presented, and the new batch concept ``meta-batch" which improves
the performance of NN several times more is introduced.
  The new activation function, meta-batch method, features of numerical data,
meta-augmentation with metaparameters, and a structure of NN generating a
compact multi-layer perceptron(MLP) are essential in this study.
I. K. Hong
# Smooth Mathematical Function from Compact Neural Networks

###### Abstract

This is paper for the smooth function approximation by neural networks (NN). Mathematical or physical functions can be replaced by NN models through regression. In this study, we get NNs that generate highly accurate and highly smooth function, which only comprised of a few weight parameters, through discussing a few topics about regression. First, we reinterpret inside of NNs for regression; consequently, we propose a new activation function-integrated sigmoid linear unit (ISLU). Then special characteristics of metadata for regression, which is different from other data like image or sound, is discussed for improving the performance of neural networks. Finally, the one of a simple hierarchical NN that generate models substituting mathematical function is presented, and the new batch concept "meta-batch" which improves the performance of NN several times more is introduced.

The new activation function, meta-batch method, features of numerical data, meta-augmentation with metaparameters, and a structure of NN generating a compact multi-layer perceptron(MLP) are essential in this study.

smooth function approximation, artificial intelligence, neural network, compactness, smoothness, activation function, batch
Introduction

In many fields, such as astronomy, physics, and economics, someone may want to obtain a general function that satisfies a dataset through regression from numerical data, which are fairly accurate ([1; 2; 3; 4]). The problem of smoothly approximating and inferring general functions using neural networks (NNs) has been considered in the some literature. However, there is insufficient research on using NNs to completely replace the ideal mathematical functions of highly smooth levels, which are sufficiently precise to be problem-free when a simulation is performed. This study aims to completely replace such ideal mathematical functions.

Assuming a model \(M(X)\) was developed by regression on a dataset using an NN. \(M(X)\) for input \(X\) can be thought of as a replacement of a mathematical function \(f(X)\). In this study, such NN is called "_neural function(NF)_" as a mathematical function created by an NN. The components of an analytic mathematical function can be analyzed using a series expansion or other methods, whereas it is difficult for a NF.

In this study, we _created "highly accurate" and "highly smooth" NFs with a "few parameters" using metadata._ Particularly, we combined _a new activation function, a meta-batch method, and weight-generating network (WGN)_ to realize the desired performances.

The major contributions of this study can be summarized as follows.

* We dissected and interpreted the middle layers of NNs. The outputs of each layer are considered basis functions for the next layer; from this interpretation, we proposed a _new activation function_-integrated sigmoid linear unit (ISLU)-suitable for regression.
* The characteristics and advantages of metadata for regression problems were investigated. A training technique with _fectious metaparameters and data augmentation_, which significantly improves performance, was introduced. It was also shown that for regression problems, _the function values at specific locations_ could be used as metaparameters representing the characteristics of a task.
* NN structures that could _generate compact_1_NFs_ for each task from metaparameters were investigated, and a new batch concept- _'meta-batch'_-that could be used in the NFs was introduced. Footnote 1: Comprising few parameters
NNs for regression

Let's talk about an easy but non-common interpretation about regression with a multilayer perceptron (MLP). What do the outputs of each layer of an MLP mean? They can be seen as _basis functions that determine the function to be input to the next layer_.

The input \(x_{i+1}\) of the (\(i+1\))th layer can be expressed as follows:

\[x_{j}^{i+1}=\sum_{k}w_{j,k}^{i}*M_{k}^{i}(x_{0})+b_{j}, \tag{1}\]

where \(x_{0}\) denotes the input of the first layer, \(w_{j,k}^{i}\) denotes the weight that connects the \(k\)th node of the \(i\)th layer to \(j\)th node of the (\(i+1\))th layer, and \(M_{k}^{i}\) denotes a model comprising the \(0\)th to \(i\)th layers and having the \(k\)th node of the \(i\)th layer as the output. This is similar to the expression \(f(x)=\sum_{j}w_{j}\phi_{j}(x)+b\) of the radial basis function(RBF) kernel method. Clearly, the outputs of each layer act as basis functions for the next layer. Figure 2 shows the outputs of each layer of an MLP that learned the dataset \(D=\{(x_{i},y_{i})|y=0.2(x-1)x(x+1.5)\), \(x\in[-2,2]\}\) with the exponential linear unit (ELU) activation function.

To efficiently extract the final function, the output functions of the intermediate layers must be well-developed. If the output functions of each layer are well-developed, the desired final NF can be compact. In addition, for the final function of NN to be infinitely differentiable, the output functions of the intermediate layers should also be infinitely differentiable.

Figure 1: Perspective on MLP

Figure 2: The output graphs of each layer, trained with an MLP, where the nodes of each layer are
If the activation function is a rectified linear unit(ReLU), the output function bends sharply after every layer. If a one-dimensional regression problem is modeled with a simple MLP that has (k+1) layers with nodes \([N_{0},N_{1},N_{2}..N_{k}]\), the output function will bend more than \(N_{0}*N_{1}...N_{k}\). The ELU activation function weakens such bending but does not smoothen it for the first derivative. Moreover, apt attention is required when using the hyperbolic tangent function for all layers in a regression problem because the output function bends in two places after each layer.

Thus, the question is which activation function can develop the intermediate basis functions well? If the activation function starts as a linear function and bends at an appropriate curvature after each layer, the final result will be good. Therefore, we propose an activation function suitable for regression, called _"integrated sigmoid linear unit(ISLU)"_.

\[\log(\alpha+exp(\beta x))/\beta-\log(1+\alpha)/\beta, \tag{2}\]

where \(\alpha\) and \(\beta\) are positive numbers.

Our experiment shows that ISLU performs sufficiently well and is worth further research. It can improve the accuracy and smoothness of our experimental data. 2 Mathematically, ISLU for \(\alpha=1\) is a translated SoftPlus that passes the origin, but ISLU absolutely differs from SoftPlus. The purposes of their production differ, and there is a significant difference in their results.3

Figure 4: Scores. The numerical score table is shown in Appendix E.
The experimental results are shown in Figure 4.567 By default, a model structure is represented in the form "[the name of the model structure]_([the number of layers]L, [the number of nodes of all hidden layers)N]_[activation function]_[further information (option)]." The experimental metadataset is described in Appendix A <1>, which has \(B\), \(k\), and \(m\) as metaparameters and the corresponding task dataset for \(L\),\(t\),\(\phi\). 8 The average of the "sum of error squares" for eight tasks among the experimental metadatasets is considered a score.

Footnote 5: In our experiment, the Swish activation function was also tested, and its performance was comparable to that of ISLU. However, for consistency, we do not discuss it in the main text; the details are presented in Appendix B.

Footnote 6: All box plots in this study are arranged in order of small scores from the top, and items wherein the box is invisible have a larger score than the shown graph range.

In Figure 4a, we consider a basic MLP structure trained on one task; WGN and fMLP, which will be introduced hereinafter, in Figure 4b,4c are models trained using metadata. Considering ISLU[0], what is in [] represents a degree of freedom in which the activation function's shape can be changed. ISLU[0] is trained with \(\alpha=0.5\) and \(\beta=1\), ISLU[1]\({}_{a}\) is trained with \(\alpha=0.5\) and \(\beta=var\), and ISLU[1]\({}_{b}\) is trained with \(\alpha=0.5\) and \(\beta=1+var\), where \(var\) are trainable parameters. Because variables tend to be learned in a distribution with a mean near zero when training with an NN, ISLU[1]\({}_{a}\) bends slightly after each layer and ISLU[1]\({}_{b}\) bends to a certain amount and additionally adjusts its degree. 9

Footnote 7: All experimental conditions of NNs in this study are shown in Appendix D

Footnote 8: Most of the experiments in this study are done with the experimental dataset.

Footnote 9: The smaller the \(\beta\) value, the closer ISLU is to a straight line.

Considering the experimental results in Figure 4, the following is observed.

* (1) There is a significant difference in performance between SoftPlus and ISLU.
* (2) Considering an MLP, there is not much difference in performance between ISLU and ELU (Figure 4a). However, in all models trained with metadata, ISLU significantly outperforms ELU (Figure 4b,4c).

Figure 5: Comparison of ELU and ISLU when training with WGN. From left to right, the 0th, 1st, and 2nd derivatives of the curves with respect to time t in a task in the given metadatasets. Blue lines:

WGN_(4L,64N)_ELU_MB, Red lines: WGN_(4L,64N)_ISLU[1]a_MB
* (3) In Figure 3(b), when the number of nodes is high(64N), ISLU[0] outperforms ISLU[1], whereas when the number of nodes is low(15N,16N), ISLU[1] outperforms ISLU[0].
* (4) In Figure 3(c), ISLU[1]\({}_{b}\) always outperforms ISLU[0].
* (5) As shown in ISLU[1]\({}_{a}\) and ISLU[1]\({}_{b}\), there are slight differences in performance depending on what the shape of ISLU is based on.

The reason for (2) can be explained as follows: setting an activation function parameter entails giving a certain bias. When given well, it considerably helps in predicting results; otherwise, it may interfere. When using metadata, the performance is improved because biases are determined by referring to various data.

We now discuss the reasons for (3) and (4). In Figure 3(b), fMLP indicates an MLP structure trained with fictitious metadata10 for only one task. If an MLP has a lots of nodes, even if the curvature functions of all activations are set to be the same, several functions can be added and combined to produce curves with the desired shapes. Meanwhile, when the nodes are few, desired curves may not be obtained well without adjusting the curvatures of the activation functions. In Figure 3(c), WGN is a network structure 11 that learns the entire metadata at once. In this case, using ISLU[1] allows the activation shape to change between tasks, yielding better results than the fixed-shaped ISLU[0].

Footnote 10: described in II.2

Footnote 11: described in III.1

The ISLU presented in this study is an example of an activation function for creating desired curves; a better activation function can be studied.

### Perspectives of Metadata

In this study, _metadata_ are the data of datasets that are sometimes the sets of task datasets, _metafeatures_ are features of a task dataset, and _metalabels_ or _metaparameters_ are parameters representing metafeatures. Consider a case where a physical system has the relation \(y=f(x_{1},x_{2}..)\) and the function \(f\) depends on the variables \(a1,a2....\). For example, a pendulum's

Figure 6: Metadata structure.
kinetic energy \(E\) is \(E=f(\theta)\), where \(\theta\) denotes the angle between the string and gravitational field direction, and the function \(f\) depends on the string's length \(l\) or pendulum's mass \(m\).

In this case, the kinetic energy \(E\) can be viewed not only as \(f(\theta,l,m..)\) but also as \(f_{l,m}(\theta)\). The dataset \(\mathcal{D}=\{(l_{i},m_{i},\theta_{i},E_{i})|E_{i}=f(\theta_{i},l_{i},m_{i}..) \}=\{(l_{i},m_{i},D_{i})|D_{i}=D_{m_{i},l_{i}}(\theta)\}\) is metadataset and the numerical value \(l,m\) can be considered as metaparameters.

One might want to interpret the kinetic energy as \(E=f_{l,\theta}(m)\). This cannot be said to be wrong, and there may be various perspectives and interpretations for _a numerical dataset used for regression_.

### Advantages of Training with Metadata and Meta-Augmentation

Consider an experiment performed with the following metadata \(\mathcal{D}_{k}=\{(x_{i},y_{i})|y_{i}=A_{k}*\sin(p_{k}*x_{i}+\phi_{k}),x\in[0, 10],A_{k}\in[-1.5,1.5],p_{k}\in[0.5,1.5],\phi_{k}\in[0,2\pi]\}\). It can be seen from the perspective that the tasks \(\mathcal{D}=\{(x_{i},y_{i})|y_{i}=A*\sin(p*x_{i}+\phi)\}\) are given according to the metaparameters of \(A\), \(p\), and \(\phi\). In this case, if not only \(x\) but also \(A\), \(p\), and \(\phi\) were trained as training inputs, a curve could be created with zero shot just by setting \(A\), \(p\), and \(\phi\). 12 Consequently, if metadata are used to learn, the accuracy of each task increases.

Footnote 12: MLP with inputs \(A,p,\phi,\theta\) and the WGN in III.1 were used for the experiment.

Taking a hint from the fact that metadata improve inference accuracy for each task, it can be thought that even in a situation where only one task is given, fictitious metadata with fictitious metalabels (or metaparameters) can be generated to learn curves. If only fictitious metalabels are used and the data remain the same, curves would be learned in the direction of ignoring the metalabels; therefore, some data modifications are required. For the experiment, fictitious metadata comprising 10 tasks with the metaparameter \(a\) were created by moving the \(y_{i}\) value in parallel \(\pm 0.05\) for every \(a=\pm 0.02\) with the original data of \(a=0\) for a given task \(\mathcal{D}=\{(x_{i},y_{i})\}\). As a result of using fictitious metadata, the score improved significantly (Figure 9). The performance improvement was similar even when the fictitious metadata were generated by moving \(x_{i}\) instead of \(y_{i}\) according to the fictitious metalabel.

We reiterate that data augmentation _including ficitous meta-parameters_ is required to achieve significant performance improvement, otherwise there is little performance improvement. In this study, only the experimental results using MLP with fictitious metaparameters added to inputs are
shown; however, further experiments show that the performance improvement due to fictitious metadata occurs independent of the model structure.

### Learning Function with Restricted Metadata

The regression task for the numerical dataset \(\mathcal{D}=\{(x_{i},y_{i})|i=0,1,2..\}\) can have a significant advantage different from the image problems, i.e., \(y_{i}\)_values at particular locations can be metaparameters that represent the entire task dataset_. For the set of images, _if we know the RGB values at specific positions of pixels, it does not help to distinguish the features of images_. However, for a set of mathematical functions f(x)s such as fifth degree polynomial or sine curve sets, _just knowing f(x) at specific x positions can let us distinguish the functions well_. This can be shown in the experiments with sine curve datasets. For the tasks \(\mathcal{D}_{k}=\{(x_{i},y_{i})|y_{i}=A_{k}*\sin(p_{k}*x_{i}+\phi_{k}),x\in[0, 10],A_{k}\in[-1.5,1.5],p_{k}\in[0.5,1.5],\phi_{k}\in[0,2\pi]\}\) that are not given metaparameters \(A\), \(p\), and \(\phi\), it is possible to learn the sine curves just using the function values \(y_{i}\) at six points of \(x_{i}\) as metaparameters (Figure 7). In other words, it is possible to perform _few-shot learning_ simply without metalabels.

In addition, the relationship between the six \(y\) points and \(A\), \(p\), and \(\phi\) can be learned with a simple MLP that has six-dimensional inputs and three-dimensional outputs, indicating that the metaparameters \(A\), \(p\), and \(\phi\) can be completely extracted to generate a sine curve using on the six
points.

## III Function-generating networks

### Wgn

When learning metadata in a regression problem, one can think of an hierarchical NN structure in which a NF corresponding to each task is generated from corresponding meta parameters. The structure in which a model is generated from variables has been studied extensively ([5; 6]). We consider the one of the structure of a function-generating network called _weight generating network(WGN)_ in this study. As shown in Figure 6, WGN generates parameters such as the weight and bias of _main network_ through a simple MLP called _weight generator_ from metaparameters. If there are trainable parameters of the activation function on the main network, can also be generated from metaparameters.

WGN is expected to generate _NFs comprising a few parameters_ corresponding to each task through the weight generator.This is because enormous data and weight generators carefully generate the parameters of the main network. Experiments showed that WGN is effective in creating the main network with excellent performance, although it comprises only a few parameters.

What are the advantages of creating a NF with _only a few parameters_? First, because the number of times that a linear input function can be bent is reduced, it may have a regulation effect or help create a smooth function. Second, it may be helpful in interpreting and analyzing the network by directly adjusting the parameters. Third, because the number of weights is small and the inference speed is fast, it can be advantageous when a fast interference speed is required, such as a simulation.

### Meta-batch

When training a function-generating network, such as WGN, 'one' metalabel (or metaparameter) \(z_{i}\) is usually placed on the weight generator's input, and it is updated with the batch of the corresponding task on the main network. However, in this case, it becomes training with batch-size=1 for the metaparameters, and when it is updated with backpropagation at once, the metacorrelation between tasks is not used well. From these problems, the _meta-batch_ concept is proposed. To distinguish the _meta-batch_ from the conventional batch, the batch of each task corresponding to
one \(z_{i}\) is called "task batch." "Meta-batch" refers to both the batch of metaparameters and the corresponding batch of the tasks. The training method for WGN using the meta-batch is as follows.

Suppose a training metadataset \(\mathcal{D}=\{(\mathcal{D}_{k},z_{k})|k\in\{1..K\}\}\) comprising task training datasets \(\mathcal{D}_{k}=\{(x_{i}^{k},y_{i}^{k})\}_{i=1}^{N_{k}}\) are given, where \(N_{k}\) is the number of datapoints of \(\mathcal{D}_{k}\) task. For index sets \(M\subset\{1,,\,,K\},T_{k}\subset\{1,,\,,N_{k}\}\) that determines meta-batch and task batch, select the batch \(\mathcal{X}_{M}=\{(D_{m},z_{m})|m\in M\}\) and \(\mathcal{X}_{T}^{M}=\{(x_{t}^{m},y_{t}^{m})|t\in T_{m},m\in M\}\).

We denote the dimensions of \(x_{i},y_{i}\), and \(z_{i}\) as \(N[x],N[y]\), and \(N[z]\), respectively. \(w_{ij}^{l}\) denotes the weight between the \(l\)th and (\(l+1\))th layers of the WGN's main network, which has a shape \((N[w_{l}],N[w_{l+1}])\), where \(N[w_{i}]\) denotes the number of nodes at the \(i\)-th layer. The inputs \(\mathcal{X}_{T}^{M}\) of the main network are rank-3 tensors in the form of \((\mathrm{MB},\mathrm{TB},N[x])\), where \(\mathrm{MB}\) and \(\mathrm{TB}\) denote the sizes of \(M\) and \(T\), respectively.

If \(z_{m}\) enters to weight generator as inputs in the form of \((\mathrm{MB},N[z])\), \(G[w_{ij}^{l}](z_{m})\) generates a tensor in the form \((\mathrm{MB},N[w_{l}]*N[w_{l+1}])\) and it is reshaped as \((\mathrm{MB},N[w_{l}],N[w_{l+1}])\), where \(G[w_{ij}^{l}]\) denotes a generator that generates \(w_{ij}^{l}\). The outputs of the \(l\)-th layer of the main network, which has the shape \((\mathrm{MB},\mathrm{TB},N[w_{l}])\), are matrix-produced with the weights in the form \((\mathrm{MB},N[w_{l}],N[w_{l+1}])\), and then it becomes a tensor in the form \((\mathrm{MB},\mathrm{TB},N[w_{l+1}])\).13 Finally, the outputs of the main network with shape \((\mathrm{MB},\mathrm{TB},N[y])\) and \(y_{t}^{m}\) are used to calculate the loss of the entire network. Conceptually, it is simple as shown in Figure 10.

Footnote 13: All other parameters in the main network can be generated from weigh generators using a similar method

As a result of the experiment, Figure 1214 shows a significant difference in performance between
using and not using meta-batch, where "MB" means using meta-batch, and "ST" means training by inputting metaparameters individually without using meta-batch. Figure 12 also shows the difference between using WGN and just using a simple MLP.

Meta-batch can be used in any function-generating network structure that generates models from variables; another example is shown in Figure 11. The outputs of generators concatenate with the layers of the main network. As a result of experimenting with ISLU[1] in the structure shown in Figure 11, there was a performance difference of more than four times between using and not using meta-batch.

Figure 13 shows the results of using WGN and meta-batch compared with those of using only MLP. "sWGN" indicates a WGN trained with metaparameters that are the function values at 10 points of \((L,t,\phi)\) without using original metaparameters "\(B\), \(k\), and \(m\)." "mMLP" indicates an MLP that trained with a six-dimensional input combined with "\(L\), \(t\), and \(\phi\)" and the original metaparameters. "MLP" indicates a trained model for each task with just inputs "\(L\), \(t\), and \(\phi\)." This figure shows that using meta-batch, WGN outperformed MLP with fewer parameters. This also shows that WGN excels at learning all metadata and using them with only a few parameters.

Figures 14 and 15 shows the results of other metadatasets, which are described in Appendix A. The combinations of ISLU, meta-batch, and WGN give much better performance than MLP in terms of accuracy and compactness.

Figure 12: Comparison between using meta-batch and not using meta-batch

Figure 13: Scores for each task of metadata from different models.
## IV Conclusion

In this study, we focus on creating mathematical functions with desired shapes using an NN with a few parameters. Irregular and numerous parameters are helpful for generalizations because of randomness; however, this sometimes makes it difficult to interpret the network and reduces the smoothness of the functions.

In this study, we dissected NNs for regression; consequently, we proposed a new activation function. We looked at the special features of regression-related metadata, such as the possibilities to extract meta-parameters immediately, and how, given only one task, we could create fictitious meta-parameters and metadata to increase performance by more than a few times.

In addition, the network structures generating NFs from metaparameters were discussed and the _meta-batch_ method was introduced and tested for the structure called WGN. WGN makes it possible to provide smooth and desired-shaped NFs comprised of a few parameters because it carefully generates different parameters and shapes of activation functions for each task.

The findings of this study, as well as the insights obtained in the process, are significant for earning smooth and accurate functions from NNs. One of them is the perspective of obtaining desired output functions at _intermediate_ layers from enormous data. Regarding regression problems, it will help elucidate how to find the metafeature of each task and map to the corresponding metaparameter as well as how to get a smooth and compact NF of a desired shape.