ANNZ is a fast and simple algorithm which utilises artificial neural networks ANNs it was known as one of the pioneers of machine learning approaches to photometric redshift estimation decades ago. We enhanced the algorithm by introducing new activation functions like tanh softplus SiLU Mish and ReLU variants its new performance is then vigorously tested on legacy samples like the Luminous Red Galaxy LRG and Stripe82 samples from SDSS as well as modern galaxy samples like the Physics of the Accelerating Universe Survey PAUS. This work focuses on testing the robustness of activation functions with respect to the choice of ANN architectures particularly on its depth and width in the context of galaxy photometric redshift estimation. Our upgraded algorithm which we named ANNZ shows that the tanh and Leaky ReLU activation functions provide more consistent and stable results across deeper and wider architectures with gt 1 per cent improvement in rootmeansquare error sigmatextrmRMS and 68th percentile error sigma68 when tested on SDSS data sets. While assessing its capabilities in handling high dimensional inputs we achieved an improvement of 11 per cent in sigmatextrmRMS and 6 per cent in sigma68 with the tanh activation function when tested on the 40narrowband PAUS dataset it even outperformed ANNZ2 its supposed successor by 44 per cent in sigmatextrmRMS. This justifies the effort to upgrade the 20yearold ANNZ allowing it to remain viable and competitive within the photoz community today. The updated algorithm ANNZ is publicly available at httpsgithub.comimdadmptANNzPlus.
['Astrophysics - Instrumentation and Methods for Astrophysics', 'Astrophysics - Cosmology and Nongalactic Astrophysics']
ANNZ an enhanced photometric redshift estimation algorithm with applications on the PAU Survey
{'ANNZ+: an enhanced photometric redshift estimation algorithm with applications on the PAU Survey': "Imdad Mahmud Pathi, a John Y. H. Soo, a Mao Jie Wee, a Sazatul Nadhilah Zakaria, a Nur Azwin Ismail, a Carlton M. Baugh, b Giorgio Manzoni, c Enrique Gaztanaga, d,e,f Francisco J. Castander, e,f Martin Eriksen, g,h Jorge Carretero, h,i Enrique Fernandez, g Juan Garcia-Bellido, j Ramon Miquel, g,k Cristobal Padilla, g Pablo Renard, l Eusebio Sanchez, i Ignacio Sevilla-Noarbe, i and Pau Tallada-Cresp'ı h,i  \n- a School of Physics, Universiti Sains Malaysia, 11800 USM, Pulau Pinang, Malaysia\n- b Institute for Computational Cosmology, Department of Physics, Science Laboratories, Durham University, South Road, Durham, DH1 3LE, UK\n- c Jockey Club Institute for Advanced Study, The Hong Kong University of Science and Technology, Hong Kong S.A.R., China\n- d Institute of Cosmology and Gravitation (ICG), University of Portsmouth, PO1 3FX Portsmouth, UK\n- e Institute of Space Sciences (ICE-CSIC), Universitat Aut'onoma de Barcelona, Carrer de Can Magrans S/N, E-08193 Cerdanyola del Vall'es (Barcelona), Spain\n- f Institut d'Estudis Espacials de Catalunya (IEEC), Edifici RDIT, Universitat Polit'ecnica de Catalunya, E-08860 Castelldefels (Barcelona), Spain\n- g Institut de F'ısica d'Altes Energies (IFAE), Universitat Aut'onoma de Barcelona, E-08193 Bellaterra (Barcelona), Spain\n- h Port d'Informaci'o Cient'ıfica (PIC), Universitat Aut'onoma de Barcelona, Carrer Albareda S/N, E-08193  \nBellaterra (Barcelona), Spain  \n- i Centro de Investigaciones Energ'eticas, Medioambientales y Tecnol'ogicas (CIEMAT), Avenida Complutense 40, E-28040 Madrid, Spain\n- j Instituto de F'ısica Te'orica UAM-CSIC, Universidad Auton'oma de Madrid, E-28049 Cantoblanco (Madrid), Spain", 'E-mail: johnsooyh@usm.my': 'Abstract. annz is a fast and simple algorithm which utilises artificial neural networks (ANNs), it was known as one of the pioneers of machine learning approaches to photometric redshift estimation decades ago. We enhanced the algorithm by introducing new activation functions like tanh, softplus, SiLU, Mish and ReLU variants; its new performance is then vigorously tested on legacy samples like the Luminous Red Galaxy (LRG) and Stripe-82 samples from SDSS, as well as modern galaxy samples like the Physics of the Accelerating Universe Survey (PAUS). This work focuses on testing the robustness of activation functions with respect to the choice of ANN architectures, particularly on its depth and width, in the context of galaxy photometric redshift estimation. Our upgraded algorithm, which we named annz+ , shows that the tanh and Leaky ReLU activation functions provide more consistent and stable results across deeper and wider architectures with > 1 per cent improvement in root-mean-square error ( σ RMS ) and 68th percentile error ( σ 68 ) when tested on SDSS data sets. While assessing its capabilities in handling high dimensional inputs, we achieved an improvement of 11 per cent in σ RMS and 6 per cent in σ 68 with the tanh activation function when tested on the 40-narrowband PAUS dataset; it even outperformed annz2 , its supposed successor, by 44 per cent in σ RMS . This justifies the effort to upgrade the 20-year-old annz , allowing it to remain viable and competitive within the photoz community today. The updated algorithm annz+ is publicly available at https://github.com/imdadmpt/ANNzPlus .', '1 Introduction': "In recent years, the astronomical community has faced a data deluge reaching exabytes due to the enormous efforts to comprehend the cosmos [1]. Numerous planned and ongoing surveys such as the Sloan Digital Sky Survey (hereafter, SDSS) [2], Dark Energy Survey (DES) [3], Euclid [4], Kilo-Degree Survey (KiDS) [5], Nancy Grace Roman Space Telescope (formerly Wide-Field Infrared Survey Telescope or WFIRST) [6], Dark Energy Spectroscopic Instrument (hereafter, DESI) [7], Subaru Prime Focus Spectrograph (PFS) [8], Hyper Suprime-Cam (HSC) [9], Legacy Survey of Space and Time (LSST) [10], and the 4-meter Multi-Object Spectroscopic Telescope (4MOST) [11] are expected to observe a significant number of galaxies in both wide and deep areas. These surveys will be able to provide highly  \naccurate measurements of the large-scale structure of the Universe, a better understanding of the formation and evolution of the galaxies changed over time and the nature of dark energy and dark matter. Additionally, brand-new resources like the James Webb Space Telescope (hereafter, JWST ) [12] allowed the first batch of galaxies with 11 < z < 20 to be observed. The latest discovery of early galaxies, using JWST medium band filters [13], highlights the crucial role of photometric redshifts in the study of the Universe's structure, and in providing candidates for follow-up spectroscopy. These surveys require good quality photometric redshifts (photoz s) to attain their research targets.  \nSpectroscopic redshifts, or specz s refer to the displacement of spectral lines measured by a spectrograph, primarily due to the expansion of the Universe. These redshifts are generally determined by analysing the galaxy's spectrum, but the process requires a lot of telescope time and uses expensive facilities. Even with the advent of spectrographs with a huge multiplex factor, like DESI which can target 5,000 objects in one pointing, deeper observations can reach much higher surface densities of galaxies than this, requiring many passes of the same area to achieve high spectroscopic completeness. This is where photoz s come in: photoz is a technique that allows us to estimate the distance to a galaxy based on its observed photometry. It is called 'photometric' because it involves measuring the amount of light emitted by a galaxy at different wavelengths. After all, the observed wavelengths of light are typically redshifted, or shifted to longer wavelengths, as the light travels through expanding space [14, 15]. Photoz s are usually estimated using broad-band magnitudes and there are two main methods: spectral template fitting and empirical/machine learning techniques. The first approach commonly employs a likelihood utilising a χ 2 fit including codes like delight [16], gazelle [17], eazy [18], zebra [19], hyperz [20], bpz [21] and le phare [22]. The second approach employs machine learning methodologies, like self-organizing maps [23], neural networks [24], deep convolutional neural networks [25], galaxy morphotoz with neural networks ( gaznets ) [26], multi-task learning networks [27], interpretable deep capsule networks [28] and bayesian neural networks [29].  \nArtificial Neural Network Redshift ( annz 1 ) introduced by [30] was among the first machine learning photometric redshift algorithms produced and widely used; it was used for photoz estimation [31, 32] concurrently with other photoz codes [33-38]. It has also been employed for morphological classification of galaxies [39-41] and star-galaxy separation [4244]. annz has been replaced by its successor annz2 2 [45] which includes some quantification of epistemic uncertainties through an ensemble of randomised estimator techniques reminiscent of modern deep ensembles. Despite the latest photoz codes, annz remains fast and simple compared to many other algorithms.  \nIn recent work, annz was used in ref. [46] as one of the 13 photometric redshift codes to assess the strengths and weaknesses of current photoz s methods for Euclid mission focusing particularly on the 0 . 2 -2 . 6 redshift range that the Euclid mission will probe in order to provide precise and accurate photoz measurements. The results showed annz was able to provide reliable single value estimates for redshifts but they faced challenges in producing useful PDF for the validation sample sources. annz also experienced sharp drops in performance metrics above redshift z ≈ 1 . 3 which it struggled with maintaining performance at higher redshifts but annz showed good results and performed generally better than the template-fitting ones in the lower redshift. Despite various machine learning algorithms, after decades, annz is still in use today.  \nHowever, ref. [47] demonstrated that artificial neural networks can yield poor results when used with high-dimensional narrowband inputs without comprehensive optimisation. Ref. [48] mentioned they have not fully explored all the possible combinations of the number of neurons and hidden layers using DLNN and CNN. This is the opportunity of this work to provide insight since annz has the ability to do it. The significant recent advances in the technology and theory of artificial neural networks raise the question: is annz still competitive 20 years after its debut?  \nOne of the most critical hyperparameters affecting the performance of an artificial neural network (hereafter, ANN) is the choice of activation function. Activation functions are mathematical functions that determine the output layer of a neuron in an ANN and transform the weighted input of a neuron into more linearly separable abstract features using multiple layers, enhancing the effectiveness of the ANN by creating non-linear relationships between variables. The most popular activation functions are the sigmoid [49], hyperbolic tangent (tanh) [50], softplus [51], rectified linear unit (ReLU) [52], leaky rectified linear unit (Leaky ReLU) [53], exponential linear unit (ELU) [54], parametric rectified linear unit (PReLU) [55], S-shaped rectified linear unit (SReLU) [56], sigmoid linear unit (SiLU) [57], and mish [58]. Activation functions can be categorised into three groups, which are fixed, adaptive and nonstandard activation functions [59, 60]. Despite using common trial and error methods to optimise ANN's hyperparameters and test various activation functions, ref. [61] did not draw any conclusions on which activation functions should be used.  \nSince annz debuted way before many of these activation functions were introduced, it only utilised one single activation function in its code: the logistic sigmoid. We were interested in testing the viability and competitiveness of annz by 'updating' it with the choice of these more up-to-date activation functions while analysing their performance as well. annz is a machine learning algorithm coded from scratch using C++, therefore it gives us the flexibility to insert and test any known activation function in the literature, and is not limited to those made available in public machine learning packages. We believe that the study of activation functions in the context of regression can significantly impact the accuracy and precision of galaxy photoz s of modern surveys, which currently still relies heavily on the synergy of results between many different algorithms and methodologies.  \nTherefore, we will address the following objectives in this work:  \n- 1. To improve the accuracy of photometric redshift estimation by identifying stable and consistent activation functions across varying architectures of artificial neural networks, and therefore assess the relevancy and competitiveness of our updated algorithm annz+ ;\n- 2. To gain valuable insights into how different activation functions respond to the complexity of high dimensional input, in the context of photometric redshift estimation.  \nWhile many may deem the change in activation function as elementary or merely one of the many ways to optimise ANNs, we emphasise that this work utilises this simple methodology to kill two birds in one stone: to improve the original annz , and to break the myth that 'ReLU is the best', which is too common a suggestion found on many online resources in regards to deep learning. We believe that the context is very important: in this case, which is of regression and photoz , we specifically chose to study the impact of activation function in regards to this specific context while keeping all other hyperparameters constant. It is hoped that this work would act as a detailed report with quantitative evidence on  \nFigure 1 . A schematic diagram of the complex inner workings of deep artificial neural networks (ANNs) with the activation function g j ( u j ) embedded in each node, highlighting their crucial role in shaping the network's behaviour and facilitating the production of efficient photoz information. The variables m 1 , ..., m n represent the inputs, w ij denote the weights, the circles symbolise the neurons, and every vertical set of circles is a layer. The number of nodes in a layer is labelled as p and q while the output is denoted as z phot . The bias node b allows the function to be shifted up or down.  \n<!-- image -->  \nANN optimisation: a resource which is relevant and significant within the current photoz community.  \nThis paper is structured as follows. In section 2, we first introduce the algorithm, activation functions and metrics used. Section 3 provides details on the data sets and training samples used. We present answers and discussions for objectives (1) and (2) in section 4, followed by a continued discussion of objective (1) in section 5. Our work concludes in section 6 with suggestions for future work. This work is a continuation and completion of the preliminary work presented in a proceeding by [62].", '2 Methodology': 'This study aims to enhance existing machine learning techniques for photoz estimation by choosing the suitable activation function. The machine learning algorithm discovers a deterministic relationship between the input variables (e.g. the ugriz broadband magnitudes) and the spectroscopic redshifts in a training set (under the assumption that the spectroscopic redshift is the true redshift) and then uses this information to produce photoz s for a target sample without any available spectroscopic redshift information. We will briefly review annz in the subsection below. Then, we will list the activation functions explored in section 2.2, followed by a description of the performance metrics used in this study in section 2.3.', '2.1 ANNZ': "Artificial Neural Network redshift estimation algorithm, known as annz [30] has been used to produce many photoz catalogues [41, 63]. annz is a multilayer perceptron (MLP) comprised of multiple layers of interconnected nodes. The input layer of nodes is the galaxy magnitudes in the available filters m = ( m 1 , m 2 , ..., m n ). The final layer is the single output node, the photometric redshift z phot . The intermediate layers are called 'hidden layers', the size and number of which are entirely up to the user. The network architecture is represented as N in : N 1 : N 2 : ... : N out , where N in is the number of input nodes, followed by N 1 in the first hidden layer, and subsequently, the number of nodes in the i th intermediate hidden layers are denoted by N i . The neural network architecture 5:10:10:1 is an example of 5 input nodes, 10 nodes in each of the two hidden layers and one output node. Each interconnecting link between the nodes is associated with a weight w ij to be optimised, where i and j describe the two nodes. An activation function, g j ( u j ), which introduce non-linearity to the model is applied at each node j and is defined as:  \nu j = [ ∑ i w ij g i ( u i ) ] + b , (2.1)  \nwhere the node u j is the value at node j that carries the weighted sum over all nodes u i to nodes j and g i ( u i ) is the activation function applied to the value u i of node i . The bias node, b , allows the function to be shifted up or down, which can be used to adjust the accuracy of the function. Figure 1 represents the building blocks of a deep ANN, which emphasises neurons producing an output by combining the linear combination of inputs and then applying a non-linear transformation known as the activation function. The original activation function used in annz is the sigmoid function.  \nThe galaxy training set needs to have photometry, m , and spectroscopic redshift, z spec , to train the ANN by adjusting the weights to minimise the cost function. The square in the cost function gives more weight to the outliers rather than taking the absolute value.  \nE = ∑ k [ z phot ( w,m k ) -z spec , k ] 2 . (2.2)  \nTo make sure the weights are regularised , an extra quadratic cost term  \nE = β ∑ i,j w 2 ij , (2.3)  \nis added to eq. (2.2): this term keeps the weight values small to avoid overfitting.  \nAn iterative quasi-Newtonian method is used in annz to execute the minimisation process (refer to [39, 64] for more details). The cost function is evaluated on a validation set after each iteration. Training stops when the cost function is minimised on the validation set to prevent overfitting. The final weights from this iteration are chosen for the ANN, which can then be used to process new input vectors and compute the outputs.  \nFor photoz , the network architecture of 5:10:10:1 was found empirically to be optimal [31, 34, 65] for MegaZ-LRG. We have set this architecture as the default setting proposed by [30]. In this study, we compare the performance of this default setting with two additional types of networks:  \n- 1. a deep network, denoted as N in : 10 d : 1, where d is the number of hidden layers, e.g. for d = 2, we have N in : 10 : 10 : 1, d = 3, we have N in : 10 : 10 : 10 : 1 and etc. We tested using up to ten hidden layers, with each layer consisting of ten nodes.\n- 2. a wide network, with a hierarchical structure denoted as N in : 10 N : 10 N : 1, where N represents an increasing number of nodes ranging from one to ten. For instance, if N = 5, the network would have 50 nodes in the first and second hidden layers (i.e., 10 × 5), giving an architecture configuration of 5:50:50:1.  \nThe method for choosing the random seed is typically via trial and error. Each committee's (or network's) training with different random seeds enhances the diversity of the learning process, thereby reducing the chances of overfitting. This approach ensures our model can generalise effectively when faced with new and unseen data. The different initial random seeds may lead to similar RMS accuracy, but slight differences of around ≲ 10 per cent can still exist [65]. However, here, some activation functions may influence the consistency of the RMS accuracy (see section 4.2 for a further discussion on this).  \nWe have chosen the default setting of the ANN to run up to a maximum of 9999 iterations; four committees were trained for each network architecture, and each was initialised with four fixed random seeds with values of [5, 50, 500, 5000]. Unfortunately, due to the randomised initialisation of the weights, two scenarios occasionally occurred: (1) annz encountered operational limitations 'NaN' and (2) annz stopped early during the training, therefore in these situations, different random seeds will be utilised.", '2.2 Activation functions': "Activation functions play an important role in determining the transformation of inputs and outputs within each layer of a neural network. However, under certain conditions, some activation functions may cause problems that significantly affect the network's performance and the stability of its training process. The most common problems include:  \n- 1. The vanishing gradient problem, which occurs when the gradients of the loss function (which are used to update the weights of the neural network during backpropagation) become very small (close to zero) as they are propagated backwards through the layers and diminish. As a result, the network has almost no gradient to update the weights and enters into a saturated region, hindering effective learning [66]. The sigmoid and tanh functions are commonly associated with this problem.\n- 2. The exploding gradient problem, which arises due to poor weight initialisation or when the gradients of the loss function become extremely large as it propagates backwards through layers, causing the weight to be updated disproportionately, leading the network to diverge or produce 'NaN' values and fail to learn anything useful [67]. Our results showed that the Softplus and Mish activation functions experience these issues when applied within complex network architectures.\n- 3. The dying gradient problem, which refers to a situation where certain neurons stop learning altogether, resulting in 'dead' neurons that contribute nothing to the model's output. This typically occurs with activation functions like ReLU, which produces zero output for all negative inputs. Consequently, the gradients of the loss function become zero during backpropagation, causing the weights to remain unchanged and preventing any further learning [68].  \nFigure 2 . The activation functions and their derivatives: sigmoid, tanh, softplus, ReLU, Leaky ReLU, SiLU and Mish. Each function is represented by a different colour: black, green, yellow, magenta, blue, orange and red, as indicated by the key.  \n<!-- image -->  \nBy understanding these issues upfront, we can analyse the impact of the activation functions on the performance of annz . Here, we summarise and review several activation functions as potential replacements for the sigmoid function originally used in annz : tanh, SiLU, softplus, ReLU, Mish, and Leaky ReLU for the benefit of photometric redshift estimation. The shapes and ranges of the activation functions are depicted in figure 2 and to encapsulate the essence of activation functions, a summary can be found in table 1.", '2.2.1 Logistic Sigmoid': "The logistic sigmoid activation function or also known as the logistic function [69] (hereafter, it will be referred to as sigmoid, which is more conventional) was used in ANN by Grossberg in 1982 [49] and was the most popular activation function in the early 1990s [70]. It is defined as  \ng j ( u j ) = 1 1 + e -u j , (2.4)  \nwith derivative  \ng ' j ( u j ) = g j ( u j )[1 -g j ( u j )] . (2.5)  \nThe function takes any real number as input and produces an output that is also a real number. As shown in figure 2, the function is differentiable and bounded to a range between 0 and 1 except for the endpoints themselves. The derivative of the sigmoid function is always positive, which makes it useful for backpropagation learning in neural networks [71].  \nAs mentioned in 2.1, annz used the sigmoid function as its default setting; however, this activation function faces significant problems during backpropagation. The vanishing gradient problem occurs when the outputs of the sigmoid function are very close to 1 or 0, causing the function to saturate when the inputs are large, either positively or negatively. Furthermore, the derivative of the sigmoid function is capped at a maximum of 0.25 (when the input is 0) and decreases rapidly as the input moves away from 0.  \n)  \n)  \nTable 1 . Summary of the activation functions, showing their functional forms, derivatives and range.  \nAdditionally, the output range of the sigmoid function (0, 1) is not centred at 0, which can lead to poor convergence [72]. Moreover, using exponential functions in the sigmoid can increase computational demands, which might slow down the overall performance of the network. While the sigmoid activation function is useful for certain tasks like classification [73], its drawbacks of vanishing and exploding gradient issues become significant in regression tasks (as evidenced by our findings) along with the limited output range and computational overhead, making it less desirable for more complex neural network architectures and tasks. It is also less effective when used in deeper networks [52].", '2.2.2 Tanh': "The hyperbolic tangent (tanh) activation function [50] defined as  \ng j ( u j ) = e u j -e -u j e u j + e -u j = 2sigmoid(2 u j ) -1 , (2.6)  \nwith derivative  \ng ' j ( u j ) = 1 -g j ( u j ) 2 , (2.7)  \nwhich is the preferred choice over the sigmoid even though they share similarities to mapping input values to an S-shaped curve, as depicted in figure 2. Besides, the tanh function has a wider range (-1, 1) than the sigmoid and it is centred at 0. Hence, this choice resolved one of the problems encountered with the sigmoid function. Due to this, the tanh function was frequently used in regression problems. However, the tanh function still faces saturated regions, and it remains exponential, which can introduce computational complexities.", '2.2.3 Softplus': "The softplus activation function [51] serves as the primitive form of the sigmoid function and is mathematically defined as  \ng j ( u j ) = ln(1 + e u j ) , (2.8)  \nwith derivative  \ng ' j ( u j ) = 1 1 + e -u j . (2.9)  \nThis is a convex function and has a non-saturating nature within the positive domain [51]. The graph of the softplus function closely resembles ReLU (see next subsection), as depicted in figure 2. Unlike the ReLU function, the softplus function exhibits slight reservations for values below 0, which can reduce the likelihood of neuronal degradation. However, it involves more computational complexity compared to the ReLU function [74]. An advantage of the softplus function lies in its smooth derivative, which proves beneficial for the backpropagation process in neural networks. Remarkably, the derivative of the softplus function is the sigmoid function. Notably, even with the ReLU function's zero-based threshold, neural networks trained using it tend to converge to local minima of equal or superior quality [68].", '2.2.4 Rectified Linear Unit (ReLU)': "ReLU [68] is a simple function,  \ng j ( u j ) = { 0 , u j < 0 u j , u j ≥ 0 , (2.10)  \ng ' j ( u j ) = { 0 , u j < 0 1 , u j ≥ 0 . (2.11)  \nwith derivative  \nIt is a widely used activation function for many types of neural networks. ReLU substitutes negative values with 0 while leaving positive values unchanged. It converges faster than those using sigmoid or tanh activation functions because ReLU is non-saturating. It does not become saturated at high input values like sigmoid and tanh, thereby effectively alleviating the vanishing gradient problem during backpropagation [52, 68]. This allows ReLU to be able to learn six times faster in large neural networks with large datasets than saturating non-linear activation functions [75-77].  \nHowever, ReLU is not centred at 0 and does not handle negative inputs well. This can cause problems during training. Leaky ReLU (section 2.2.5) is an extension of ReLU that addresses these issues.", '2.2.5 Leaky ReLU': "Leaky ReLU [53] is an improved version of ReLU, specifically designed to address the dying ReLU problem (neuron weights not changing) and has all the advantages of ReLU. Leaky ReLU is defined as  \nwith derivative  \ng j ( u j ) = { αu j , u j < 0 u j , u j ≥ 0 , (2.12)  \ng ' j ( u j ) = { α, u j < 0 1 , u j ≥ 0 . (2.13)  \nThe unique feature of Leaky ReLU is the addition of the parameter α , which represents the small slope in the u j < 0 region known as the leakage in the function. However, with this addition, the parameter α requires a value to be specified. In the majority of instances in literature, α = 0.01 is used [60].", '2.2.6 Sigmoid Linear Unit (SiLU)': "SiLU was introduced by Hendrycks and Gimpel in 2016 [57]. The function of SiLU is  \ng j ( u j ) = u j · Φ( u j ) = u j · 1 2 [ 1 + erf ( u j -µ σ √ 2 )] , (2.14)  \nwhere Φ( u j ) represents the cumulative distribution function of a Gaussian distribution and erf stands for the error function and parameters µ = 0 and σ = 1. SiLU can also be written as  \ng j ( u j ) = u j · σ ( u j ) = u j 1 + e -u j , (2.15)  \nand its derivative is,  \ng ' j ( u j ) = 1 + e -u j + u j e -u j (1 + e -u j ) 2 , (2.16)  \nwhere the sigmoid function σ ( u j ) = 1 / (1 + e -u j ) is used as an approximation to the cumulative distribution function of a Gaussian distribution.  \nFigure 2 shows SiLU are unbounded above and bounded below and distinguished by its smoothness and non-monotonicity. In fact, its non-monotonicity sets it apart from most typical activation functions as it has a global minimum value of g j ( u j ) ≈ -0 . 28 for u j ≈ -1 . 28. The global minimum, where the derivative is zero serves as an implicit regularisation that inhibits the learning of weights of large magnitudes [78]. Furthermore, being unbounded above aids in avoiding the saturation gradient problem and its smoothness also helps in the optimisation and generalisation of machine learning models [57]. All these properties make SiLU perform much better than other activation functions in reinforcement learning. SiLU is also known as Swish or Sigmoid Shrinkage [79, 80].", '2.2.7 Mish': "Mish is derived from the author's name [58] and is defined as  \ng j ( u j ) = ( u j ) · tanh[ln(1 + e u j )] , (2.17)  \nwith derivative  \ng ' j ( u j ) = sech 2 [softplus( u j )] · silu( u j ) + g j ( u j ) u j . (2.18)  \nAt first sight, Mish looks similar to SiLU, as can be observed in figure 2. Mish is smooth, non-monotonic (bounded below and unbounded above), self-regularised and continuously differentiable to avoid singularities. The softplus function in the definition of Mish ensures that the output of the function is always greater than or equal to -1. In image classification and object detection tasks, Mish consistently outperforms ReLU and SiLU between 1 to 3 per cent across all standard architectures [58].", '2.3 Photometric redshift performance metrics': 'We assess the photoz results obtained from our new annz+ using several performance metrics. This section provides the definitions for all the metrics used to quantify the performance of point estimate photoz s produced after testing for each activation function.', '1. Root-mean-square error, σ RMS ,': 'σ RMS = √ √ √ √ 1 N N ∑ i ∆ z 2 i . (2.19)  \nThe difference between the photometric and spectroscopic redshift, scaled by 1 + z spec , is ∆ z i = ( z phot ,i -z spec ,i ) / (1 + z spec ,i ). It is important to note that σ RMS is calculated without outliers removed. Therefore, it measures the overall scatter of the sample.', '2. 68th percentile error, σ 68 ,': 'σ 68 = Q 84 . 1% (∆ z i ) -Q 15 . 9% (∆ z i ) 2 , (2.20)  \nis defined as the half width of the weighted distribution of ∆ z i containing 68 per cent of the objects, where Q are the quantiles of the distribution of ∆ z i . This metric is centralised and assesses the width of the core in the photoz distribution, with the sensitivity to outliers greatly reduced.  \n- 3. Outlier fraction, η out , is the percentage of objects for which  \n| ∆ z i | ≥ 0 . 15 . (2.21)  \nThis metric [81] identifies the percentage of objects with large outliers.', '3 Data': 'In order to study the performance of activation functions towards the accuracy of photoz s, annz requires two types of data in each sample: spectroscopic (true) redshifts and multiband photometric data. The data was obtained from two surveys: SDSS (LRG and Stripe-82 are the special subsets within the larger SDSS project) and PAUS. The LRG and Stripe82 datasets utilised five broad bands ugriz along with SDSS spectroscopic redshifts. In contrast, the PAUS dataset utilised PAUS-GALFORM catalogue, which contains six broad bands ( ugri ∗ zy ∗ ) 3 and 40 narrow bands, with spectroscopic redshifts from the simulation. The data is divided into three samples: LRG, Stripe-82 and PAUS-GALFORM samples. All samples of galaxies are used to test the activation functions to get the general results and for specific purposes: the PAUS-GALFORM sample used to test the effectiveness of  \nFigure 3 . Distribution of i -band magnitude (blue histogram) and spectroscopic redshift (red histogram) for the LRG, Stripe-82 and PAUS-GALFORM samples.  \n<!-- image -->  \nthe activation functions on high dimensional input training, while the LRG and Stripe-82 samples are used for photoz comparison with other works.  \nWe define the testing set as the set of data without redshift information, where the aim is to determine the photoz s; the performance metrics in eqs. (2.19), (2.20) and (2.21) are evaluated using this set; the training set consists of available and reliable specz s information of galaxies; while the validation set is used as part of the training process to prevent overtraining. The total number of galaxies in the samples is divided equally into a 1:1:1 ratio of training-validating-testing sets as suggested by [38]. Sections 3.1, 3.2 and 3.3 describe the sources of the data for this study as well as the training samples used in this paper. The distribution of galaxies is illustrated in figure 3.', '3.1 SDSS LRG Sample': 'The SDSS luminous red galaxy (hereafter, LRG) [83] sample is selected using colour and magnitude; the LRGs are characterised by their red colour, high luminosity and faint apparent magnitude due to the presence of the strong 4000 ˚ Abreak despite lower S/N. LRGs are typically massive elliptical galaxies at intermediate redshifts (0 . 4 ≤ z ≤ 0 . 7). The LRGs provided the clearest early observation of baryon acoustic oscillations [84]. The MegaZ-LRG catalogue [31, 34] is an example of an LRG catalogue that relied on photoz .  \nWe applied photometric cuts as detailed in [83] and cross-match with photometry from the SDSS Catalog Archive Server Jobs System (CasJobs) DR12 [85]. We obtained spectroscopic data for the LRG sample through the Legacy Survey also via SDSS CasJobs. To do this, we used the flag TARGET=GALAXY RED and removed redshifts of low quality by setting  \nzWarning=0 . We then selected objects with redshifts between 0 . 1 < z < 1, zErr>0 , and magnitudes between 20 . 0 < u < 28 . 0, 19 . 5 < g < 24 . 5, 18 . 0 < r < 22 . 2, 17 . 3 < i < 20 . 9, and 16 . 7 < z < 21 . 0. This generated a sample of 412,138 galaxies.', '3.2 SDSS Stripe-82 Sample': 'The SDSS Stripe-82 Survey [86] is a compilation of imaging data (co-addition) [87], obtained by repeatedly scanning along the equatorial stripe with the following coordinates: -50 · ≤ RA ≤ 60 · and -1 . 25 · ≤ DEC ≤ 1 . 25 · . The survey reaches approximately two magnitudes fainter than the standard SDSS observations, with an approximate magnitude limit of i ∼ 24 . 1. We opted to work with objects in the Stripe-82 region because SDSS provide photometric data and plenty of spectroscopic redshift data. The wide-angle deep imaging data from this region has been widely used in various studies, such as photoz estimation [35, 38, 88], quasar classification [89], massive galaxy evolution [37], the creation of deeper co-adds [90], radio emission in radio-quiet quasars [91] and active galactic nuclei [92].  \nThe photometric data of this area can be obtained from the SDSS CasJobs, with the setting run=106,206 . It is important to note that any runs other than these are either data prior to co-addition or were not observed under photometric conditions. The following are the selection criteria for this sample: for the photometry, only objects from run=106,206 (co-added photometry), the galaxies with redshift between 0 < z < 1, magnitudes is between 17 . 0 < u < 31 . 0, 16 . 0 < g < 26 . 0, 16 . 0 < r < 24 . 5, 15 . 6 < i < 24 . 0 and 15 . 2 < z < 24 . 0 are used. Outlier and bad data has been removed from the sample, based on the following criteria: δu < 5 . 0, δg < 1 . 0, δr < 0 . 3, δi < 0 . 3, and δz < 1 . 0.  \nThe spectroscopic data in this sample use z spec within the Stripe-82 region from SDSS spectrograph [2] which have 9 × 10 5 galaxies with Petrosian magnitude r ∼ 17 . 77. To ensure only quality redshifts are used, we select all redshifts that have zWarning=0 and require spectral class=="GALAXY" . We have utilised the SDSS star-galaxy separator TYPE=3 for our sample, which helped us filter out non-extended objects. After applying these selection criteria, a sample of 72,763 galaxies was obtained.', '3.3 PAUS-GALFORM Sample': "The Physics of the Accelerating Universe Survey (hereafter, PAUS) is a narrowband photometric galaxy survey which used the 4.2 m William Herschel Telescope (WHT) at the Roque de los Muchachos Observatory (ORM) on the Canary island of La Palma, Spain [93, 94]. PAUS employed the PAUCam instrument using 40 narrowband filters ranging between 4500 and 8500 ˚ A with a spacing of 100 ˚ A, complemented with six standard broad bands [95]. The survey has successfully observed over 51 deg 2 of the sky since the beginning of 2016. When using the bcnz2 , it is possible to achieve σ 68 / (1 + z ) = 0 . 0037 for i AB < 22 . 5 in the redshift range 0 < z < 1 . 2, when selecting the best 50 per cent of the sources based on a photometric redshift quality cut [96].  \nHere, we utilise the synthetic dataset from the PAUS GALFORM mock catalogue [97], which consists of 14,053 galaxies. The dataset was simulated to mimic the observational characteristics of PAUS using the GALFORM semi-analytical model of galaxy formation within the Planck Millennium N-body [98]. By simulating the galaxy population to produce realistic properties such as positions, luminosities, colours, and redshifts, the PAUS-GALFORM mock catalogue serves as a valuable tool for studying galaxy evolution, validating models, and gaining insights into the physical processes shaping galaxies in the Universe. The lightcone mock catalogue is built by estimating galaxy brightness and positions by interpolating  \nsmoothly between output redshift from the Planck Millennium N-body simulation, using the point at which a galaxy crosses the observer's lightcone (see ref. [99, 100] for further details). The mock catalogue for PAUS covers approximately 100 deg 2 and is magnitude limited to i = 22 . 5, covering a redshift range 0 < z < 2.  \nIn our case, we did not use any photometric errors from [101], nor did we introduce any errors in the model galaxy photometry, as done by [97], as we do not want the errors to affect the performance of the ANN alone. However, we acknowledge that the photoz generated from the PAUS-GALFORM sample is constructed based on a limited number of initial parameters and synthetic magnitudes, this idealisation could allow annz+ to learn the underlying parameterisation of the semi-analytic model. This is unlike observed data such as the SDSS LRG and Stripe-82 samples, which have added real-world variability. By leveraging both mock and observed datasets, we provide a balanced assessment that highlights the robustness of the activation functions across different data scenarios and ensure that the conclusions drawn from the mock data are not overly dependent on its idealised characteristics. The photometric data has been bifurcated into two distinct samples: the PAUS-GALFORM(6), hereafter referred to as the PG-06 sample, which uses six broad bands ugri ∗ zy ∗ and the PAUS-GALFORM(46), we label as the PG-46 sample, which utilises both broad bands and the 40 narrow bands. The six bands will be used to compare the performance of the activation functions along with LRG and Stripe-82 samples, while 46 bands were used to test the impact of the activation function on high dimensional inputs.", '4 Impact of activation functions on photometric redshift performance': 'In this section, our primary focus is not on obtaining the absolute global minimum value with the best performance metrics because the most common approach to achieve the absolute global minimum is through trial and error since specific hyperparameters may not work for every sample. Instead, we are more interested in ensuring the consistency and stability of annz+ under any condition and making it suitable for universal use. We have tested various activation functions in annz+ to observe their behaviour on different samples, various network architectures, random seed variations, and the effectiveness of each activation function.', '4.1 General results': "Figure 4 shows our results when testing deep and wide networks for the LRG, Stripe-82, and PAUS-GALFORM samples. The 68th percentile error for tanh, ReLU, and Leaky ReLU consistently demonstrated stable trends for all the samples, whether deep (more layers) or wide (more neurons or nodes per layer). When we delve into a comparison between sigmoid, Mish, softplus, and SiLU in a deep network, it becomes apparent that the error increases significantly as we deepen the network architecture, i.e. as we add more 'hidden' layers. On the other hand, increasing the number of nodes per layer gives us a contrasting outcome, with the results turning out to be more stable than the deep networks. However, the processing time for wide networks is long. For instance, the computational runtime required to process the largest sample used in this work (i.e. the LRG sample) was 150 minutes for the 5:10 5 :1 architecture, while it took 547 minutes for the 5:50:50:1 architecture. The LRG sample contains 412,138 objects, leading to a mean runtime per object for each architecture to be approximately 0.02 seconds and 0.08 seconds, respectively. In the context of high-dimensional input data, the PG-46 sample took a maximum runtime of 36 minutes for deep networks and  \nTable 2 . Relative improvement (indicate by ∆%) in root-mean-square error ( σ RMS ), 68th percentile error ( σ 68 ) and outlier fraction η out for the LRG, Stripe-82 and PAUS-GALFORM (PG) samples, with respect to architecture 5:10:10:1 and sigmoid activation function. The table below only shows the top 6 results, where we see that the tanh, ReLU, and Leaky ReLU activation functions have been shown to produce competitive results. PG-06 utilises the ugri ∗ zy ∗ broad bands while PG-46 incorporates both broad bands and 40 narrow bands.  \nDeep                                                                                    Wide  \nFigure 4 . Relation between the number of hidden layers (deep network) as well as the number of nodes in each hidden layer (wide network) using LRG, Stripe-82, PG-06 and PG-46 samples corresponds to the σ 68 for different activation functions as indicated by different colour lines. The pattern of the line plot was used to determine the performance of each activation function. Tanh, Leaky ReLU and ReLU consistently have the lowest performance metric, which is considered the best activation function.  \n<!-- image -->  \n150 minutes for wide networks, utilising the mentioned architecture. The average runtime per object, across a total of 14,053 objects was 0.15 seconds (deep) and 0.64 seconds (wide). Consequently, the increased number of inputs resulted in a longer average runtime per object.  \nAfter testing with several activation functions, we found that the 5:10:10:1 architecture is enough for the tanh function to achieve the same convergence as the sigmoid function on the LRG sample. A complex architecture is unnecessary for the LRG sample. Adding extra hidden layers to a network architecture leads to greater performance gains compared to widening existing layers [65]; however, for any data set, there will be a fundamental limit to this gain: this limit is determined by the inherent variation in the data set, representing the best possible accuracy that can be achieved with the given data. The required network complexity and size of the training set will depend on this limit, as well as other factors, such as the signal-to-noise ratio.  \nEven though the sigmoid function can still provide a reasonable result when using simple networks, problems arise when dealing with deep networks and high-dimensional input (see figure 4). We will explain in more detail about high-dimensional input in section 4.5. As we can see in figure 4, the sigmoid function's performance degrades from the 7th layer onwards for the Stripe-82 and PAUS-GALFORM samples. The function reaches saturation region (as shown in figure 2) at the larger and smaller input, causing a smaller output range between 0 and 1. As the backpropagation process continues, the gradients progressively shrink until they vanish, leaving the initial hidden layers without gradient information. This is known as the vanishing and exploding gradient problem, and it is difficult for neural networks to learn and optimise effectively. Thus, it is evident that annz will need to be updated with more stable activation functions.  \nAfter analysing the overall situation, we narrowed our focus in table 2 to examine the individual activation functions in more detail. The table summarises the relative improvement for the top 6 architectures with lowest values of σ RMS , σ 68 and outlier fractions for LRG, Stripe-82 and PAUS-GALFORM samples. The ∆% indicates the improvement with respect to the performance metric of default setting: architecture, N in : 10 : 10 : 1 and sigmoid activation functions. Upon conducting tests on the LRG samples, we observed that the values for all metrics using tanh, Leaky ReLU, ReLU and sigmoid with different architectures were consistent with the default setting. We did not notice any significant improvement due to the proper selection cut of the data, which is the redshift range is small, luminosity and colour of the objects are similar. Overall, not much variety in this dataset.  \nRegarding the Stripe-82 sample, it can be concluded that using tanh does not improve σ RMS by more than 1 per cent. However, it is worth noting that tanh has a correlation with improving 68th percentile error, showing an improvement of 1.3 to 3.2 per cent. Meanwhile, by widening the existing layers, 4.5 per cent improvements of σ 68 can be achieved when using ReLU, followed by Leaky ReLU (4.0 per cent). These points suggest that tanh, ReLU, and Leaky ReLU are suitable for the σ 68 performance metric in wider networks.  \nIn the PG-06 sample, using tanh, sigmoid, ReLU and Leaky ReLU significantly improved the results. We observed a notable correlation of the metrics by utilising the tanh activation function with three hidden layers. This resulted in a 10.2 per cent of σ RMS , a 3.1 per cent σ 68 , and a 7.4 per cent improvement in the outlier fraction. The activation functions ReLU and Leaky ReLU have been found to significantly improve the accuracy of σ 68 in this sample, with a maximum improvement of 8.7 per cent. This finding is consistent with the trend observed with the Stripe-82 sample, highlighting the effectiveness of these activation functions in reducing the sensitivity to outliers in Stripe-82 and PAUS-GALFORM samples. Due to the lack of galaxies at higher redshift, it is necessary to use a deep network or add more hidden layers to the ANNs to improve the learning process.  \nOn the other hand, we found that it is not advisable to use Mish, SiLU and softplus activation functions in photoz estimation due to their poor performance, despite their ability to outperform ReLU in other problems. In this particular study, we found out that no particular architecture will work for every sample indicating that the use of the most recently published activation function is not necessarily the best choice.", '4.2 Sensitivity to random seed': "Figures 5, 6, and 7 show that each activation function has a different level of sensitivity towards the choice of the random seed that determines the weight initialisation through these three samples represented by error bars. The error bars denote the standard error,  \nLRG  \nFigure 5 . A comparison of seven activation functions-tanh, Leaky ReLU, ReLU, sigmoid, Mish, SiLU, and softplus-across varying numbers of hidden layers for the LRG sample. Each point represents the σ 68 value for a specific activation function with error bars. As the model complexity increases, we observe distinct trends for each function.  \n<!-- image -->  \nwhich is derived from the mean of σ 68 across four committees (i.e. four choices of random seed) for each architecture. Therefore, the error bars indicate the extent of fluctuation in random seed for specific activation functions.  \nAcross all samples, the optimisation of tanh and Leaky ReLU is not influenced by the random seed. Meanwhile, Mish, SiLU and softplus are very sensitive; thus, without a proper selection of random seed, the error may go up by 89 per cent because of the outliers when we cycle through the SiLU, softplus and Mish activation functions. Therefore in certain runs, we tweaked and changed some of the values of random seeds in order to maintain consistent results across each set of 4 committees as sometimes they can't converge because of the exploding gradient. The default random seeds have to be changed for the PG-46 sample with 46 bands as well, due to its high number of inputs. Particularly for wide and very deep networks, the running for softplus and Mish stops at a certain number of nodes, and some runs could never complete due to the random seed that caused them to be unable to converge. Therefore, we advise users to always test out different random seeds every time the activation functions are changed to avoid problems arising when running the ANN.  \nTheoretically, the hyperbolic tangent function does not solve the vanishing and explod-  \nStripe-82  \nFigure 6 . A comparison of seven activation functions-tanh, Leaky ReLU, ReLU, sigmoid, Mish, SiLU, and softplus-across varying numbers of hidden layers for the Stripe-82 sample. Each point represents the σ 68 value for a specific activation function with error bars. As the model complexity increases, we observe distinct trends for each function.  \n<!-- image -->  \ning gradient problems, but in annz+ , it works rather well. Despite several studies claiming that SiLU, Mish, and softplus may solve the exploding gradient problem, they end up not converging in some of our runs. The ReLU and sigmoid activation functions were found to be least sensitive to random seeds, and only a few runs showed significant error bars. For sigmoid, ReLU, Mish, softplus and SiLU, a proper selection of random seed is needed in ANN to ensure fast convergence. If the initial weights are too small, then the signal flowing through the network gradually diminishes, eventually leading to a loss of information and potentially causing the network fail to learn effectively; if the weights are initialised with values too large, the variance of input data into each layer increases rapidly and soon overflows, leading to unstable training [72].  \nAfter analysing the three different samples with varying galaxy distributions and redshift ranges, we can conclude that the fluctuation of error bars for the mean of the four committees, as shown in figures 5, 6, and 7 remained consistent and have the same patterns for all samples. We related this pattern to the landscape of the loss function. The loss landscape 4 is the geometric landscape formed by the loss function or cost function. Each activation function  \nPAUS-GALFORM (6 bands vs 46 bands)  \nFigure 7 . A comparison of seven activation functions-tanh, Leaky ReLU, ReLU, sigmoid, Mish, SiLU, and softplus-across varying numbers of hidden layers for the PAUS with six ugri ∗ zy ∗ broad bands sample. Each point represents the σ 68 value for a specific activation function with error bars. As the model complexity increases, we observe distinct trends for each function.  \n<!-- image -->  \nhas a distinct characteristic that impacts the shape and structure of the loss landscape. A comparison between the loss landscape of ReLU and Mish from the previous study reveals that ReLU functions have sharp minima, which are characterised by steep and narrow regions around the minimum [58]. ReLU introduces sparsity by setting negative values to zero, which can influence the density of the gradients in the loss landscape. On the other hand, Mish has a wider minimum, which results in a smoother and flatter loss landscape surface which contributes to better generalisation. Flat minima may lead to slower convergence during optimisation, as the gradients around flat minima are less informative and may require more iterations for optimisation.  \nIn the context of softplus and SiLU functions, the loss landscape visualisation may contain multiple local minima, saddle points, plateaus, and steep valleys, which makes it more sensitive to the random seed and difficult to optimise. Tanh and Leaky ReLU are similar to ReLU but have sharper, steeper gradients in the shape of the loss landscape, which makes the convergence continuously differentiable across their domains. This indicates a more direct path to the global minimum of the loss function.  \nTable 3 . Comparison of runtime for deriving photoz estimates from a LRG sample containing 412,138 galaxies, alongside the performance metrics ( σ RMS and σ 68 ) for various activation functions using a 5:10:10:1 network architecture.", '4.3 Time efficiency': 'In terms of effectiveness, ReLU and Leaky ReLU are the most friendly functions with respect to time, volume of data and computational efficiency. According to table 3, these simple functions can converge faster and run all of the training data (sample sizes up to 400 000) in 27 minutes with competitive σ RMS and σ 68 values. Meanwhile, networks with sigmoid and tanh activation functions converge more slowly with respective runtimes of 43 and 56 minutes for simple architecture. A longer runtime is expected when the complexity of the network increases, especially in deeper and wider networks. Mish took the longest runtime to converge with a total time of 60 minutes due to its complex functional forms, and unfortunately, also produced the most problematic results, followed by SiLU and Softplus. In our case, we need a faster point estimator that can converge faster during optimisation. It is confirmed that sharp minima are beneficial in terms of optimisation speed and efficiency [102]. This point gives credit to the use of ReLU and Leaky ReLU activation functions for managing extensive data sets associated with large-scale surveys like Euclid and LSST, which aim to calculate photoz for billions of objects.', '4.4 Comparison between ReLU variants': 'Here, the tanh function is shown to be robust and is suggested to be the best activation to be used for regression problems like estimating photoz s. While the ReLU activation function is widely used in neural networks due to its effectiveness in handling vanishing gradient problems and accelerating convergence, we find that it is not always necessary to use the ReLU activation function, particularly in regression analysis. The choice of activation function should be carefully considered depending on the specific requirements of the model and the objectives of the analysis.  \nTo a certain extent, ReLU experiences the dying ReLU problem, in which ReLU completely blocks any input less than zero. In some of our training runs, the bias becomes very negative, hence causing the neurons to fail to learn during backpropagation. The inputs are not positive enough to overcome the bias and lead to the creation of many dead neurons. To solve this problem, we suggest the use of Leaky ReLU, ELU or GeLU. In our study, we can see that Leaky ReLU slightly outperforms ReLU in σ RMS and σ 68 , as shown in table 4.', '4.5 Performance of activation functions for high-dimensional inputs': "The PAUS-GALFORM mock galaxy sample is an excellent sample to test our implementation of different activation functions for the photoz s in a high dimensional space since it consists  \nPG-46  \nFigure 8 . Plots of photoz versus specz , comparing the photoz performance when using tanh, ReLU, Leaky ReLU, sigmoid, SiLU, softplus and Mish activation functions in annz for the PG-46 sample (trained and tested using 6 ugri ∗ zy ∗ broad bands and 40 narrow bands). Comparing activation functions based on the lowest σ 68 for each activation function across various architectures to evaluate the performance. A deep network, denoted as N in : 10 d : 1, where N in is the number of input and d is the number of hidden layers. As seen from the plots, tanh, ReLU and Leaky ReLU functions showed promising results when producing photoz s with a large number of inputs. The blue lines are the limit for outliers  \n<!-- image -->  \nTable 4 . A comparison of the performance of different activation functions-tanh, SiLU, softplus, ReLU, Mish, and Leaky ReLU with respect to sigmoid for photoz s on the PAUS-GALFORM data with eight hidden layers, trained with 46 bands.  \nof 40 narrow bands and six broadband filters. We vary the choice of activation functions to gain valuable insights into how different activation functions respond to the complexity and high dimensionality of the PAUS data. From table 2, we see that the tanh function in a deep network when testing with 46 bands brings about up to 4.0 per cent improvement in σ RMS to photoz with respect to the PG-06 sample with the default setting of annz . This indicates that the tanh function is helpful in enabling ANNs to estimate photoz from 46 bands. This improvement can potentially solve the issue raised in ref. [30], where additional input could lead to a decline in the performance of ANNs.  \nUpon comparison of performance between the PG-46 sample with respect to the default setting, it was observed that utilising deeper architecture and the tanh activation function resulted in a notable improvement of up to 8.5 per cent in σ RMS , 14.9 per cent in σ 68 and 12.9 per cent improvement of the outlier fraction. By adding more hidden layers, ReLU and Leaky ReLU also achieved better performance. This indicates that changing the activation function and utilising deeper architecture can help to reduce the impact of the high dimensionality and achieve significant improvements in performance than changing the architecture alone. This could solve the problem as mentioned in ref. [47].  \nIt was found that deep networks potentially performed well for the PAUS data, therefore, we fixed the architecture to 46:10:10:10:10:10:10:10:10:1 to assess the impact of deep network architecture on each activation function. Table 4 tabulates the results of each activation function's performance on this sample with respect to the sigmoid activation function. The results show that using tanh activation functions in high-dimensional inputs and deep networks can lead to a 79.7 per cent improvement in σ RMS , 88.5 per cent in σ 68 and 98.1 per cent in the outlier fraction compared to sigmoid, followed by Leaky ReLU. The difference in σ RMS (34.1 per cent) and σ 68 (41.6 per cent) between tanh and ReLU proves that ReLU is not always the best even though ReLU is the most popular activation function in the literature. Despite the improvements found with softplus, SiLU, and Mish, the performance metrics exhibit a significant range (0.1219 to 0.1572), and the presence of an outlier fraction which deviates from the norm.  \nIt is evident that the sigmoid function failed and did not facilitate effective learning in the ANNs, as it did yield the highest performance metric. This is due to the derivative of the sigmoid activation function becomes saturated (close to zero) for large positive and large negative inputs (see figure 2). From eq. (2.2), the derivative of the error function concerning the weights and biases is proportional to the product of the derivatives of the activation functions of each layer. Taking an eight-layer network with sigmoid activation as  \nTable 5 . A comparison of three photo-z estimatorsannz (sigmoid), annz2 (sigmoid and tanh), and annz+ (tanh)-across different neural network architectures: default, deep, and wide. The table includes improvement percentage σ RMS and σ 68 for each estimator on LRG, Stripe-82 and PAUSGALFORM samples. The performance of ANNz+ shows a significant improvement compared to other estimators.  \nan example, the derivative of the error function with respect to the weight w ij connecting the i -th neuron in the seven-layer to the j -th neuron in the eight-layer is  \n∂E ∂w (7) ij ∝ g ' [ u (8) j ] g ' [ u (7) i ] , (4.1)  \nwhere u (8) j and u (7) i are the weighted inputs of the neurons on the eighth and seventh layers, respectively. If the values of u (8) j and u (7) i are very large or very small, then g ' [ u (8) j ] g ' [ u (7) i ] will be very close to zero (see sigmoid activation function derivatives in figure 2), and so will ∂E ∂w (7) ij .  \nThis means that the weight w (7) ij will not be updated significantly during the backpropagation algorithm, and the network will not learn effectively. The ANNs converge before the limit and are unable to improve beyond that. The same problem applies to biases as well. This illustrates the vanishing gradient phenomenon, which makes the learning process very slow and can prevent the neural network from reaching the optimal weights and biases.  \nThe scatter plot in figure 8 displays the correlation between photoz and specz with the diagonal line representing perfect correlation. Comparing activation functions based on the lowest σ 68 for each activation function across various architectures to evaluate the performance. Tanh aligns closely along the diagonal, approaching accurate predictions. ReLU and Leaky ReLU also shows good alignment, but the galaxies scatter more at higher redshifts. Sigmoid, SiLU, softplus and Mish exhibit mediocre performance. This statistical analysis demonstrates that using sigmoid, softplus, SiLU and Mish with high-dimensional inputs is not recommended. These photoz results will be compared with delight , bcnz2 and cigale in an upcoming paper (Soo et al. in prep).", '5.1 Comparison of ANNZ+ with ANNZ2': "We are interested in comparing the performance of annz+ and annz2 when the ANN hyperparameters are fixed, such as sample sizes, network architecture, and activation functions.  \nFigure 9 . Plots of photo-z versus spec-z, comparing the results of the σ 68 of annz (blue), annz+ (green) and annz2 (red) for all samples.  \n<!-- image -->  \nannz2 [45] is the successor to annz , which differs in programming language and functionality despite their similar names. The annz2 algorithm is a powerful machine learning based photoz estimator that uses a mixture of different machine learning methods, including artificial neural networks, boosted decision trees, and k -nearest neighbours, to produce probability density functions (PDFs) that provide highly accurate photoz point estimates. This algorithm uses the Toolkit for Multivariate Data Analysis ( tmva ) [103] with root [104], which allows it to run multiple different machine learning algorithms for training and output photoz s based on a weighted average of their performance. annz2 calculate uncertainties using the k -Nearest Neighbors ( k NNs) method and has been widely used in recent works [38, 88, 105] due to its high customisability and ability to produce accurate results.  \nHere, annz2 version 2.3.1 was run using artificial neural networks individually optimised for best performance. The configurations used were NCycles=9999 , EstimatorType=MSE , TrainingMethod=BFGS , LearningRate=0.02 and UseRegulator=False . We evaluated the performance of annz2 using the same set of samples as presented in section 3. annz2 comes with both sigmoid and tanh activation functions, thus we used annz (utilised sigmoid function) and annz+ (utilised tanh function) to be compared with annz2 's results. To this end, we considered three distinct network architectures, which are N in : 10 : 10 : 1 (default), N in : 10 : 10 : 10 : 10 : 10 : 10 : 10 : 10 : 10 : 10 : 1 (deep) and N in : 50 : 50 : 1 (wide). For LRG and Stripe-82 samples, there are N in = 5 inputs, while for PAUS samples, N in = 6 and 46 inputs were used.  \nThe mean value of the PDF ( z pdf ) was chosen to represent the photoz point estimate for annz2 , which is calculated via a weighted average across all runs. This is because ref. [38] found that z pdf produces better overall accuracy at the high redshift end as compared to the peak (mode) value of the PDF ( z peak ). However, if the PDF has a low redshift contribution, we acknowledge that z peak may represent the most suitable value that could yield different results from annz2 . Moreover, regardless of the point estimate method chosen, whether the mean or modeannz+ consistently outperforms annz2 . It achieves a 3.6 per cent improvement in the LRG sample, a 9.5 per cent in the Stripe-82 sample, and a 25.9 per cent in the PG-46 sample. These comparisons remain valid for annz2 when it employs results generated from z peak , utilised tanh activation function and adheres to the default architecture. The purpose of this investigation was to obtain insight into the impact of network architecture on the overall performance of annz2 .  \nThe improvement percentages of photoz estimation results of these runs for annz and annz+ as compared to annz2 are summarised in table 5 and visualised in the figure 9. The overall result shows annz+ performs much better than annz2 . As we can see, the performance metrics of annz2 on σ RMS and σ 68 are higher than annz+ , resulting in poorer performance of annz2 compared to annz+ with difference up to 2.5 per cent for the LRG sample, 12.1 per cent for Stripe-82, 40.6 per cent for the PG-06 sample and 43.6 per cent difference for the PG-46 sample despite changes in the activation function. Surprisingly, even though annz2 uses the sigmoid function, it still works well in deep networks compared to the original annz , which gives worse results. Furthermore, the use of tanh has been shown to enhance the accuracy of photoz s in annz2 . Figure 9, shows annz+ gets many of the high specz objects correctly close to the diagonal line of the scatter plot for Stripe-82, PG-06 and PG-46. This shows that annz+ , despite showing not much difference in results for the LRG sample, has shown promising results to produce better photoz s for high redshift samples. As we move to the annz2 , it is evident that annz2 does not perform well when dealing with high dimensional inputs where the scatter plot density of galaxies did not focus on the  \nTable 6 . Comparison of the photometric redshift performance of other works were used to compare with our results ( annz+ ) for the testing set of the MegaZ-LRG catalogue and Stripe-82 survey. The performance metrics with lower σ RMS and σ 68 indicate better accuracy.  \nregression line.  \nThe discrepancies between both algorithms are due to the regularisation method, backpropagation uncertainty generation method and the optimisation process in between annz and annz2 . Ref. [47] showed that annz2 performed poorly in high-dimensional input when a large number of inputs is used, which aligns with our findings. The utilisation of k NNs by annz2 to generate errors might be the underlying reason, it looks at the nearby objects in the significantly high-dimensional ugri ∗ zy ∗ +40 narrowband space and uses them to determine the associated uncertainties. We can see that the k NN method probably works the best when the number of inputs is small, as the curse of dimensionality may have diluted the separating power of k NNs to determine better uncertainties. Despite differences in architecture, it is worth noting that annz+ demonstrate a higher level of stability than their predecessors. We also recognise that annz2 was designed to generate accurate PDFs, thus it may not be optimised for producing derived point estimates. These observations provide strong justification for updating annz to ensure that it remains relevant and competitive alongside annz2 .", '5.2 Crossmatch with MegaZ-LRG catalogue and Stripe-82 Survey photometric redshifts': 'We crossmatch our annz+ (tanh) results from the LRG sample with the MegaZ-LRG catalogue [34] and a few other works, summarising the results in table 6. We can see that the results between annz , annz2 and annz+ are consistent up to a difference of 0.001 in σ RMS . We note that this result is expected since the LRG sample has a very stringent cut in terms of colour, redshift range and luminosity. We have shown that the results of annz+ remain the lowest in terms of σ RMS and most competitive in its results, overtaking the results of one  \nof the most recent methods which is a combination of annz2 and bpz [88]. In term of σ 68 , annz2 x bpz improved the accuracy by 0.001 compared with annz+ .  \nFinally, we evaluate the photoz performance for the Stripe-82 Survey as well. The photoz results of annz+ outperform morphoz , annz2 using tanh and those by ref. [35] for all metrics. We note however that morphoz have been weighted in accordance to the densities of the target sample and made use of spectroscopic redshift data from non-SDSS sources as well, thus its result is notably different (refer to ref. [38] for more details)', '6 Conclusions and future work': "We aimed to improve the accuracy of annz , a 20-year-old photometric redshift algorithm, by selecting an activation function to optimise the artificial neural network and give the most stable and consistent results. In the context of photometric redshift estimation and regression, it can be concluded that the tanh function is the most stable and competitive activation function all the time. It consistently produces reliable results, unaffected by random seeds or different types of architectures. For the LRG sample, the tanh function yields consistent σ RMS and σ 68 values with the previous literature, approximately 0.0281 and 0.0228 followed by the Leaky ReLU and ReLU activation function. In the Stripe-82 sample, the tanh function improves the accuracy of σ 68 by 3.5 per cent when using a wider network, along with < 1 per cent improvement of the σ RMS using the default architecture. In the PAUS-GALFORM samples, the tanh function showed the lowest performance metrics, resulting in a 10.7 per cent improvement in σ RMS and a 5.7 per cent enhancement in σ 68 for the PG-06 sample, as well as an 8.5 per cent improvement in σ RMS and a 14.9 per cent enhancement in σ 68 for the PG-46 sample.  \nWe also found out that the tanh function works well in photoz estimation when using high dimensional inputs by alleviating the vanishing and exploding gradient problems faced by the sigmoid function when going too deep network architectures. We got up to a 4.0 per cent improvement of σ RMS for PAUS-GALFORM 46 bands. This indicates that the tanh function is helpful in enabling ANNs to estimate photoz from 46 bands. When we fixed the architecture and compared the performance across different activation functions, tanh achieved a 79.7 per cent, followed by Leaky ReLU with a 78.9 per cent difference compared to the sigmoid function.  \nAt this point, ReLU and Leaky ReLU are suitable for wide networks and to get the absolute best for σ 68 as they can get approximately 8.7 per cent improvement on PAUSGALFORM 6 bands and 4.0 - 4.5 per cent for Stripe-82 samples. The ReLU and Leaky ReLU activation functions facilitate an efficacious learning process, characterised by their capability to expedite the completion of the training process. On the other hand, it is not advisable to use Mish, SiLU and softplus activation functions in photoz estimation due to their poor performance in our results, despite their ability to outperform ReLU in other machine learning problems. Sigmoid, SiLU, softplus, and Mish are all very sensitive to random seeds; therefore, caution is needed when selecting random seeds.  \nWe introduced annz+ as an improved version of annz by replacing the sigmoid activation function with the tanh activation function as default. To assess the relevancy and competitiveness of annz+ , we compared it with annz2 using the tanh activation function and found a maximum difference in the performance of 43.6 per cent. A similar improvement can be seen in annz when compared with annz2 using a sigmoid function, except when applied to deep and wide network architecture in annz . We crossmatch the MegaZ-LRG  \ncatalogue and Stripe-82 Survey with the other works, annz+ consistently outperforms the other methods.  \nWe recommend the photoz community to use tanh when dealing with regression analysis because most of the newest activation functions generally only do well in classification problems. To ensure optimal performance across all conditions, it is imperative to identify the most suitable activation function. This involves a thorough analysis of various factors that impact the function's efficacy, such as the type of data being processed and the specific task at hand. By carefully selecting the appropriate activation function, one can achieve optimal results and enhance the overall efficiency of the system. For normal regression analysis, we find that it is not necessary for the ANN to go too deep as it is not worth the additional training time, while if we want to get the absolute best, we can attempt going deep using the robust tanh function.  \nFurther investigation into whether annz+ is consistently better than annz2 is needed. As it stands, the point photoz estimate with annz+ is indeed better. However, upcoming surveys such as Euclid and LSST will probe higher redshifts and the complexity of photometric redshift estimation will increase due to redshift degeneracies, such as the Lyman/Balmer break degeneracy, which becomes more prominent for samples spanning 2 < z < 3. In such cases, algorithms that are capable of producing PDFs rather than point estimates are likely to perform better and thus are more valued by the community. While annz+ currently focuses on point estimates, it might be possible to modify annz+ to produce PDFs and see if it can compete with annz2 . To implement PDF generation within the annz+ framework, several modifications are necessary. First, we need to write additional C++ scripts to locate the output layer configuration, allowing for the production of either Gaussian parameters or binned probabilities. Next, we should define the loss function to align with these modifications. Finally, we will need to create additional scripts to plot the PDF. This could be a good project to explore in the future.", 'Acknowledgments': "IMP would like to thank Joseph Ooi Boon Han and Yoon Tiem Leong, for the fruitful discussions. JYHS acknowledges financial support via the Fundamental Research Grant Scheme (FRGS) by the Malaysian Ministry of Higher Education with code FRGS/1/2023/STG07/ USM/02/14. JYHS and IMP acknowledge financial support from the Short Term Grant by Universiti Sains Malaysia with code 304/PFIZIK/6315395. IMP also acknowledges financial assistance under the Gra-Assist 2020 scheme provided by the Institute of Postgraduate Studies (IPS), Universiti Sains Malaysia.  \nCMB acknowledges support from the Science Technology Facilities Council through ST/X001075/1. GM is supported by the Collaborative Research Fund under Grant No. C6017-20G which is issued by the Research Grants Council of Hong Kong S.A.R. EG acknowledges grants from Spain Plan Nacional (PGC2018-102021-B-100) and Maria de Maeztu (CEX2020-001058-M). FJC acknowledges grants from Spain Plan Nacional (PID2022-141079NBC31) and Maria de Maeztu (CEX2020-001058-M). JC acknowledges support from the grant PID2021-123012NA-C44 funded by MCIN/AEI/ 10.13039/501100011033 and by 'ERDF A way of making Europe'.  \nCP acknowledges support from the Spanish Plan Nacional project PID2021-123012NBC41. JGB acknowledges grants from Spain Plan Nacional (PID2021-123012NB-C43) and Centro de Excelencia Severo Ochoa (CEX2020-001007-S). RM acknowledges support from the  \nSpanish Plan Nacional project PID2021-123012NB-C41. CP acknowledges support from the Spanish Plan Nacional project PID2019-111317GB-C32 and PID2022-141079NB-C32. PR acknowledges the support by the Tsinghua Shui Mu Scholarship, the funding of the National Key R&D Program of China (grant no. 2023YFA1605600), the National Science Foundation of China (grant no. 12073014 and 12350410365), the science research grants from the China Manned Space Project with No. CMS-CSST2021-A05, and the Tsinghua University Initiative Scientific Research Program (No. 20223080023). ES acknowledges grants from Spain Plan Nacional (PID2021-123012NB-C42). ISN acknowledges grants from Spain Plan Nacional (PID2021-123012NB-C42). ME acknowledges funding by MCIN with funding from European Union NextGenerationEU (PRTR-C17.I1) and by Generalitat de Catalunya.  \nFunding for the SDSS-I/II/III has been provided by the Alfred P. Sloan Foundation, the Participating Institutions, the National Science Foundation, the U.S. Department of Energy, the National Aeronautics and Space Administration, the Japanese Monbukagakusho, the Max Planck Society, and the Higher Education Funding Council for England. The SDSS-I/II web site is http://www.sdss.org/, while the SDSS-III web site is http://www.sdss3.org/.  \nSDSS-I/II/III is managed by the Astrophysical Research Consortium for the Participating Institutions. The Participating Institutions include the American Museum of Natural History, Astrophysical Institute Potsdam, University of Basel, University of Cambridge, Case Western Reserve University, University of Chicago, Drexel University, Fermilab, the Institute for Advanced Study, the Japan Participation Group, Johns Hopkins University, the Joint Institute for Nuclear Astrophysics, the Kavli Institute for Particle Astrophysics and Cosmology, the Korean Scientist Group, the Chinese Academy of Sciences (LAMOST), Los Alamos National Laboratory, the Max-Planck-Institute for Astronomy (MPIA), the Max Planck-Institute for Astrophysics (MPA), New Mexico State University, Ohio State University, University of Pittsburgh, University of Portsmouth, Princeton University, the United States Naval Observatory, the University of Washington, University of Arizona, the Brazilian Participation Group, Brookhaven National Laboratory, Carnegie Mellon University, University of Florida, the French Participation Group, the German Participation Group, Harvard University, the Instituto de Astrofisica de Canarias, the Michigan State/Notre Dame/JINA Participation Group, Lawrence Berkeley National Laboratory, Max Planck Institute for Extraterrestrial Physics, New York University, Pennsylvania State University, the Spanish Participation Group, University of Tokyo, University of Utah, Vanderbilt University, University of Virginia, and Yale University.  \nPAUS is partially supported by the Ministry of Economy and Competitiveness (MINECO, grants CSD2007-00060, AYA2015-71825, ESP2017-89838, PGC2018-094773, PGC2018-102021, SEV-2016-0588, SEV-2016-0597 and MDM-2015-0509). Funding for PAUS has also been provided by Durham University (ERC StG DEGAS259586), ETH Zurich, and Leiden University (ERC StG ADULT279396). The PAU data centre is hosted by the Port d'Informaci'o Cien'ıfica (PIC), maintained through a collaboration of CIEMAT and IFAE, with additional support from Universitat Aut'onoma de Barcelona and the European Research Development Fund (ERDF).", 'References': "- [1] J.V. Rodr'ıguez, I. Rodr'ıguez-Rodr'ıguez and W.L. Woo, On the application of machine learning in astronomy and astrophysics: A text-mining-based scientometric analysis , Wiley Interdiscip. Rev.: Data Min. Knowl. Discov. 12 (2022) e1476.  \n- [2] D.G. York, J. Adelman, J.J.E. Anderson, S.F. Anderson, J. Annis, N.A. Bahcall et al., The Sloan Digital Sky Survey: Technical Summary , Astron. J. 120 (2000) 1579.\n- [3] . The Dark Energy Survey Collaboration, The Dark Energy Survey , preprint (arXiv:astro-ph/0510346) (2005) .\n- [4] R. Laureijs, J. Amiaux, S. Arduini, J.L. Augu'eres, J. Brinchmann, R. Cole et al., Euclid Definition Study Report , preprint (arXiv:1110.3193) (2011) .\n- [5] J.T.A. de Jong, G.A. Verdoes Kleijn, K.H. Kuijken, E.A. Valentijn, KiDs, and Astro-WISE Consortiums,, The Kilo-Degree Survey , Exp. Astron. 35 (2013) 25.\n- [6] D. Spergel, N. Gehrels, J. Breckinridge, M. Donahue, A. Dressler, B.S. Gaudi et al., Wide-Field InfraRed Survey Telescope-Astrophysics Focused Telescope Assets WFIRST-AFTA Final Report , preprint (arXiv:1305.5422) (2013) .\n- [7] M. Levi, C. Bebek, T. Beers, R. Blum, R. Cahn, D. Eisenstein et al., The DESI experiment, a whitepaper for snowmass 2013 , preprint (arXiv:1308.0847) (2013) .\n- [8] M. Takada, R.S. Ellis, M. Chiba, J.E. Greene, H. Aihara, N. Arimoto et al., Extragalactic science, cosmology, and galactic archaeology with the subaru prime focus spectrograph , Publ. Astron. Soc. Pac. 66 (2014) .\n- [9] H. Aihara, N. Arimoto, R. Armstrong, S. Arnouts, N.A. Bahcall, S. Bickerton et al., The Hyper Suprime-Cam SSP Survey: Overview and survey design , Publ. Astron. Soc. Pac. 70 (2017) .\n- [10] v. Ivezi'c, S.M. Kahn, J.A. Tyson, B. Abel, E. Acosta, R. Allsman et al., LSST: From Science Drivers to Reference Design and Anticipated Data Products , Astrophys. J. 873 (2019) 111.\n- [11] R.S. de Jong, O. Agertz, A.A. Berbel, J. Aird, D.A. Alexander, A. Amarsi et al., 4MOST: Project overview and information for the First Call for Proposals , The Messenger 175 (2019) 3.\n- [12] H. Yan, Z. Ma, C. Ling, C. Cheng and J.S. Huang, First Batch of z ≈ 11 -20 Candidate Objects Revealed by the James Webb Space Telescope Early Release Observations on SMACS 0723-73 , Astrophys. J. Lett. 942 (2023) L9.\n- [13] S. Withers, A. Muzzin, S. Ravindranath, T.E. Ghassan Sarrouh, R. Abraham, Y. Asada et al., Spectroscopy from photometry: A population of extreme emission line galaxies at 1 . 7 ≲ z ≲ 6 . 7 selected with jwst medium band filters , Astrophys. J. Lett. 958 (2023) L14.\n- [14] D.C. Koo, Optical multicolors : a poor person's Z machine for galaxies , Astron. J. 90 (1985) 418.\n- [15] E.D. Loh and E.J. Spillar, Photometric Redshifts of Galaxies , Astrophys. J. 303 (1986) 154.\n- [16] B. Leistedt, J. Alsing, H. Peiris, D. Mortlock and J. Leja, Hierarchical bayesian inference of photometric redshifts with stellar population synthesis models , Astrophys. J. Suppl. Ser. 264 (2023) 23.\n- [17] R. Kotulla and U. Fritze, Impact of subsolar metallicities on photometric redshifts , Mon. Not. Roy. Astron. Soc. 393 (2009) L55.\n- [18] G.B. Brammer, P.G. van Dokkum and P. Coppi, EAZY: A Fast, Public Photometric Redshift Code , Astrophys. J. 686 (2008) 1503.\n- [19] R. Feldmann, C.M. Carollo, C. Porciani, S.J. Lilly, P. Capak, Y. Taniguchi et al., The Zurich Extragalactic Bayesian Redshift Analyzer and its first application: COSMOS , Mon. Not. Roy. Astron. Soc. 372 (2006) 565.\n- [20] M. Bolzonella, J.M. Miralles and R. Pell'o, Photometric redshifts based on standard SED fitting procedures , Astron. Astrophys. 363 (2000) 476.  \n- [21] N. Ben'ıtez, Bayesian Photometric Redshift Estimation , Astrophys. J. 536 (2000) 571.\n- [22] S. Arnouts, S. Cristiani, L. Moscardini, S. Matarrese, F. Lucchin, A. Fontana et al., Measuring and modelling the redshift evolution of clustering: the Hubble Deep Field North , Mon. Not. Roy. Astron. Soc. 310 (1999) 540.\n- [23] R. Buchs, C. Davis, D. Gruen, J. DeRose, A. Alarcon, G.M. Bernstein et al., Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing , Mon. Not. Roy. Astron. Soc. 489 (2019) 820.\n- [24] J. Lee and M.S. Shin, Estimation of Photometric Redshifts. I. Machine-learning Inference for Pan-STARRS1 Galaxies Using Neural Networks , Astron. J. 162 (2021) 297.\n- [25] J. Pasquet, E. Bertin, M. Treyer, S. Arnouts and D. Fouchez, Photometric redshifts from SDSS images using a convolutional neural network , Astron. Astrophys. 621 (2019) A26.\n- [26] R. Li, N.R. Napolitano, H. Feng, R. Li, V. Amaro, L. Xie et al., Galaxy morphoto-Z with neural Networks (GaZNets) , Astron. Astrophys. 666 (2022) A85.\n- [27] L. Cabayol, M. Eriksen, J. Carretero, R. Casas, F.J. Castander, E. Fern'andez et al., The pau survey and euclid: Improving broadband photometric redshifts with multi-task learning ⋆ , Astron. Astrophys. 671 (2023) A153.\n- [28] B. Dey, B.H. Andrews, J.A. Newman, Y.Y. Mao, M.M. Rau and R. Zhou, Photometric redshifts from SDSS images with an interpretable deep capsule network , Mon. Not. Roy. Astron. Soc. 515 (2022) 5285.\n- [29] E. Jones, T. Do, B. Boscoe, J. Singal, Y. Wan and Z. Nguyen, Improving photometric redshift estimation for cosmology with lsst using bayesian neural networks , Astrophys. J. 964 (2024) 130.\n- [30] A.A. Collister and O. Lahav, ANNz: Estimating Photometric Redshifts Using Artificial Neural Networks , Publ. Astron. Soc. Pac. 116 (2004) 345.\n- [31] A. Collister, O. Lahav, C. Blake, R. Cannon, S. Croom, M. Drinkwater et al., MegaZ-LRG: a photometric redshift catalogue of one million SDSS luminous red galaxies , Mon. Not. Roy. Astron. Soc. 375 (2007) 68.\n- [32] D.G. Bonfield, Y. Sun, N. Davey, M.J. Jarvis, F.B. Abdalla, M. Banerji et al., Photometric redshift estimation using Gaussian processes , Mon. Not. Roy. Astron. Soc. 405 (2010) 987.\n- [33] H. Hildebrandt, S. Arnouts, P. Capak, L.A. Moustakas, C. Wolf, F.B. Abdalla et al., PHAT: Photo-z Accuracy Testing , Astron. Astrophys. 523 (2010) A31.\n- [34] F.B. Abdalla, M. Banerji, O. Lahav and V. Rashkov, A comparison of six photometric redshift methods applied to 1.5 million luminous red galaxies , Mon. Not. Roy. Astron. Soc. 417 (2011) 1891.\n- [35] R.R.R. Reis, M. Soares-Santos, J. Annis, S. Dodelson, J. Hao, D. Johnston et al., The sloan digital sky survey co-add: A galaxy photometric redshift catalog , Astrophys. J. 747 (2012) 59.\n- [36] C. S'anchez, M. Carrasco Kind, H. Lin, R. Miquel, F.B. Abdalla, A. Amara et al., Photometric redshift analysis in the Dark Energy Survey Science Verification data , Mon. Not. Roy. Astron. Soc. 445 (2014) 1482.\n- [37] K. Bundy, A. Leauthaud, S. Saito, A. Bolton, Y.T. Lin, C. Maraston et al., The stripe 82 massive galaxy project. i. catalog construction , Astrophys. J. Suppl. Ser. 221 (2015) 15.\n- [38] J.Y.H. Soo, B. Moraes, B. Joachimi, W. Hartley, O. Lahav, A. Charbonnier et al., Morpho-z: improving photometric redshifts with galaxy morphology , Mon. Not. Roy. Astron. Soc. 475 (2018) 3613.\n- [39] O. Lahav, A. Nairn, L.J. Sodr'e and M.C. Storrie-Lombardi, Neural computation as a tool for galaxy classification: methods and examples , Mon. Not. Roy. Astron. Soc. 283 (1996) 207.  \n- [40] N.M. Ball, J. Loveday, M. Fukugita, O. Nakamura, S. Okamura, J. Brinkmann et al., Galaxy types in the Sloan Digital Sky Survey using supervised artificial neural networks , Mon. Not. Roy. Astron. Soc. 348 (2004) 1038.\n- [41] M. Banerji, F.B. Abdalla, O. Lahav and H. Lin, Photometric redshifts for the Dark Energy Survey and VISTA and implications for large-scale structure , Mon. Not. Roy. Astron. Soc. 386 (2008) 1219.\n- [42] E. Bertin and S. Arnouts, Sextractor: Software for source extraction , Astron. Astrophys. Suppl. Ser. 117 (1996) 393.\n- [43] M.T. Soumagnac, F.B. Abdalla, O. Lahav, D. Kirk, I. Sevilla, E. Bertin et al., Star/galaxy separation at faint magnitudes: application to a simulated dark energy survey , Mon. Not. Roy. Astron. Soc. 450 (2015) 666.\n- [44] E.J. Kim and R.J. Brunner, Star-galaxy classification using deep convolutional neural networks , Mon. Not. Roy. Astron. Soc. 464 (2016) 4463.\n- [45] I. Sadeh, F.B. Abdalla and O. Lahav, ANNz2: Photometric Redshift and Probability Distribution Function Estimation using Machine Learning , Publ. Astron. Soc. Pac. 128 (2016) 104502.\n- [46] Euclid Collaboration, G. Desprez, S. Paltani, J. Coupon, I. Almosallam, A. Alvarez-Ayllon et al., Euclid preparation , Astron. Astrophys. 644 (2020) A31.\n- [47] J.Y.H. Soo, B. Joachimi, M. Eriksen, M. Siudek, A. Alarcon, L. Cabayol et al., The PAU Survey: narrow-band photometric redshifts using Gaussian processes , Mon. Not. Roy. Astron. Soc. 503 (2021) 4118.\n- [48] E. Euclid Collaboration, L. Bisigello, C.J. Conselice, M. Baes, M. Bolzonella, M. Brescia et al., Euclid preparation: XXIII. Derivation of galaxy physical properties with deep machine learning using mock fluxes and H-band images , preprint (arXiv:2206.14944) (2022) .\n- [49] S. Grossberg, How does a brain build a cognitive code? , in Studies of Mind and Brain: Neural Principles of Learning, Perception, Development, Cognition, and Motor Control , (Dordrecht), pp. 1-52, Springer Netherlands (1982).\n- [50] Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard et al., Backpropagation Applied to Handwritten Zip Code Recognition , Neural Comput. 1 (1989) 541.\n- [51] C. Dugas, Y. Bengio, F. B'elisle, C. Nadeau and R. Garcia, Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference , MIT Press (2001).\n- [52] V. Nair and G.E. Hinton, Rectified linear units improve restricted boltzmann machines , in ICML'10 , (Madison, WI, USA), pp. 807-814, Omnipress, 2010.\n- [53] A.L. Maas, A.Y. Hannun and A.Y. Ng, Rectifier nonlinearities improve neural network acoustic models , in Proc. ICML. WDLASL. , vol. 30, (Atlanta, Georgia, USA), p. 3, 2013.\n- [54] D.A. Clevert, T. Unterthiner and S. Hochreiter, Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , preprint (arXiv:1511.07289) (2015) .\n- [55] K. He, X. Zhang, S. Ren and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , in Proc. IEEE. ICCV. , (Santiago, Chile, USA), pp. 1026-1034, December, 2015.\n- [56] X. Jin, C. Xu, J. Feng, Y. Wei, J. Xiong and S. Yan, Deep Learning with S-Shaped Rectified Linear Activation Units , Proc. AAAI 30 (2016) .\n- [57] D. Hendrycks and K. Gimpel, Gaussian error linear units (gelus) , preprint (arXiv:1606.08415) (2016) .\n- [58] D. Misra, Mish: A self regularized non-monotonic activation function , preprint (arXiv:1908.08681) (2019) .  \n- [78] S. Elfwing, E. Uchibe and K. Doya, Sigmoid-weighted linear units for neural network function approximation in reinforcement learning , Neural networks 107 (2018) 3.\n- [79] P. Ramachandran, B. Zoph and Q.V. Le, Searching for activation functions , preprint (arXiv:1710.05941v2) (2017) .\n- [80] A.M. Atto, D. Pastor and G. Mercier, Smooth sigmoid wavelet shrinkage for non-parametric estimation , IEEE ICASSP (2008) 3265.\n- [81] O. Ilbert, S. Arnouts, H.J. McCracken, M. Bolzonella, E. Bertin, O. Le F'evre et al., Accurate photometric redshifts for the cfht legacy survey calibrated using the vimos vlt deep survey , Astron. Astrophys. 457 (2006) 841.\n- [82] T. Erben, H. Hildebrandt, L. Miller, L. van Waerbeke, C. Heymans, H. Hoekstra et al., Cfhtlens: the canada-france-hawaii telescope lensing survey - imaging data and catalogue products , Mon. Not. Roy. Astron. Soc. 433 (2013) 2545.\n- [83] D.J. Eisenstein, J. Annis, J.E. Gunn, A.S. Szalay, A.J. Connolly, R.C. Nichol et al., Spectroscopic target selection for the sloan digital sky survey: The luminous red galaxy sample , Astron. J. 122 (2001) 2267.\n- [84] D.J. Eisenstein, I. Zehavi, D.W. Hogg, R. Scoccimarro, M.R. Blanton, R.C. Nichol et al., Detection of the baryon acoustic peak in the large-scale correlation function of sdss luminous red galaxies , Astrophys. J. 633 (2005) 560.\n- [85] S. Alam, F.D. Albareti, C.A. Prieto, F. Anders, S.F. Anderson, T. Anderton et al., The eleventh and twelfth data releases of the sloan digital sky survey: Final data from sdss-iii , Astrophys. J. Suppl. Ser. 219 (2015) 12.\n- [86] J. Annis, M. Soares-Santos, M.A. Strauss, A.C. Becker, S. Dodelson, X. Fan et al., The sloan digital sky survey coadd: 275 deg 2 of deep sloan digital sky survey imaging on stripe 82 , Astrophys. J. 794 (2014) 120.\n- [87] L. Jiang, X. Fan, F. Bian, I.D. McGreer, M.A. Strauss, J. Annis et al., The sloan digital sky survey stripe 82 imaging data: Depth-optimized co-adds over 300 deg2 in five filters , Astrophys. J. Suppl. Ser. 213 (2014) 12.\n- [88] I.Y.K. Alshuaili, J.Y.H. Soo, M.Z. Mat Jafri and Y. Rafid, Improving photometric redshifts by merging probability density functions from template-based and machine learning algorithms* , Astron. Lett. 48 (2022) 665.\n- [89] C.M. Peters, G.T. Richards, A.D. Myers, M.A. Strauss, K.B. Schmidt, v. Ivezi'c et al., Quasar classification using color and variability , Astrophys. J. 811 (2015) 95.\n- [90] J. Fliri and I. Trujillo, The iac stripe 82 legacy project: a wide-area survey for faint surface brightness astronomy , Mon. Not. Roy. Astron. Soc. 456 (2015) 1359.\n- [91] M. Liao, J. Wang, W. Kang and M. Zhou, Exploring the connection between ultraviolet/optical variations and radio emission in radio-quiet quasars: clues about the origin of radio emission , Mon. Not. Roy. Astron. Soc. 512 (2022) 296.\n- [92] R. Carvajal, I. Matute, J. Afonso, R.P. Norris, K.J. Luken, P. S'anchez-S'aez et al., Selection of powerful radio galaxies with machine learning , Astron. Astrophys. 679 (2023) A101.\n- [93] N. Ben'ıtez, E. Gazta˜naga, R. Miquel, F. Castander, M. Moles, M. Crocce et al., Measuring baryon acoustic oscillations along the line of sight with photometric redshifts: The pau survey , Astrophys. J. 691 (2009) 241.\n- [94] F.J. Castander, Galaxies and cosmology with the projects physics of the accelerating universe (pau) and dark energy survey (des) , in SF2A-2011: Proceedings of the Annual meeting of the French Society of Astronomy and Astrophysics , G. Alecian, K. Belkacem, R. Samadi and D. Valls-Gabaud, eds., (Paris), pp. 69-72, 2011.  \n- [95] C. Padilla, F.J. Castander, A. Alarc'on, J. Aleksic, O. Ballester, L. Cabayol et al., The physics of the accelerating universe camera , Astron. J. 157 (2019) 246.\n- [96] M. Eriksen, A. Alarcon, E. Gaztanaga, A. Amara, L. Cabayol, J. Carretero et al., The pau survey: early demonstration of photometric redshift performance in the cosmos field , Mon. Not. Roy. Astron. Soc. 484 (2019) 4200.\n- [97] G. Manzoni, C.M. Baugh, P. Norberg, L. Cabayol, J.L. van den Busch, A. Wittje et al., The PAU Survey: a new constraint on galaxy formation models using the observed colour redshift relation , Mon. Not. Roy. Astron. Soc. 530 (2024) 1394 [ 2311.10469 ].\n- [98] C.M. Baugh, V. Gonzalez-Perez, C.D.P. Lagos, C.G. Lacey, J.C. Helly, A. Jenkins et al., Galaxy formation in the Planck Millennium: the atomic hydrogen content of dark matter haloes , Mon. Not. Roy. Astron. Soc. 483 (2019) 4922 [ 1808.08276 ].\n- [99] A.I. Merson, C.M. Baugh, J.C. Helly, V. Gonzalez-Perez, S. Cole, R. Bielby et al., Lightcone mock catalogues from semi-analytic models of galaxy formation - I. Construction and application to the BzK colour selection , Mon. Not. Roy. Astron. Soc. 429 (2013) 556 [ 1206.4049 ].\n- [100] L. Stothert, P. Norberg, C.M. Baugh, A. Alarcon, A. Amara, J. Carretero et al., The PAU Survey: spectral features and galaxy clustering using simulated narrow-band photometry , Mon. Not. Roy. Astron. Soc. 481 (2018) 4221 [ 1807.03260 ].\n- [101] L. Cabayol, M. Eriksen, A. Amara, J. Carretero, R. Casas, F.J. Castander et al., The pau survey: estimating galaxy photometry with deep learning , Mon. Not. Roy. Astron. Soc. 506 (2021) 4048.\n- [102] L. Dinh, R. Pascanu, S. Bengio and Y. Bengio, Sharp minima can generalize for deep nets , in Proceedings of the 34th International Conference on Machine Learning , D. Precup and Y.W. Teh, eds., pp. 1019-1028, PMLR, 2017.\n- [103] A. Hoecker, P. Speckmayer, J. Stelzer, J. Therhaag, E. von Toerne, H. Voss et al., Tmva-toolkit for multivariate data analysis , preprint (arXiv:physics/0703039) (2007) .\n- [104] R. Brun and F. Rademakers, Root - an object oriented data analysis framework , Nucl. Instr. Meth. Phys. Res. 389 (1997) 81.\n- [105] S.J. Schmidt, A.I. Malz, J.Y.H. Soo, I.A. Almosallam, M. Brescia, S. Cavuoti et al., Evaluation of probabilistic photometric redshift estimation approaches for the rubin observatory legacy survey of space and time (lsst) , Mon. Not. Roy. Astron. Soc. 499 (2020) 1587."}