Artificial neural networks revolutionized many areas of computer science in recent years since they provide solutions to a number of previously unsolved problems.
 On the other hand, for many problems, classic algorithms exist, which typically exceed the accuracy and stability of neural networks.
 To combine these two concepts, we present a new kind of neural networks—algorithmic neural networks (AlgoNets).
 These networks integrate smooth versions of classic algorithms into the topology of neural networks.
 Our novel reconstructive adversarial network (RAN) enables solving inverse problems without or with only weak supervision. Artificial Neural Networks are employed to solve numerous problems, not only in computer science but also in all other natural sciences. Yet, the reasoning for the topologies of neural networks seldom reaches beyond empirically-based decisions. In this work, we present a novel approach to designing neural networks-algorithmic neural networks (short: AlgoNet). Such networks integrate algorithms as algorithmic layers into the topology of neural networks. However, propagating gradients through such algorithms is problematic, because crisp decisions (conditions, maximum, etc.) introduce discontinuities into the loss function. If one passes from one side of a crisp decision to the other, the loss function may change in a non-smooth fashion-it may "jump." That is, the loss function suddenly improves (or worsens, depending on the direction) without these changes being locally noticeable anywhere but exactly at these "jumps." Hence, a gradient descent based training, regardless of the concrete optimizer, cannot approach these "jumps" in a systematic fashion, since neither the loss function nor the gradient provides any information about these "jumps" in any place other than exactly the location at which they occur. Therefore, a smoothing is necessary, such that information about the direction of improvement becomes exploitable by gradient descent also in the area surrounding the "jump." That is, by smoothing, e.g., an if, one can smoothly, by gradient descent, undergo a transition between the two crisp cases using only local gradient information. Generally, for end-to-end trainable neural network systems, all components should at least be C 0 smooth, i.e., continuous, to avoid "jumps." However, having C k smooth, i.e., k times differentiable and then still continuous components with k ≥ 1 is favorable. This property of higher smoothness allows for higher-order derivatives and thus prevents unexpected behavior of the gradients. Hence, we designed smooth approximations to basic algorithms where the functions representing the algorithms are ideally C ∞ smooth. That is, we designed pre-programmed neural networks (restricted to smooth components) with the structure of given algorithms. Related work [1] - [3] in neural networks focused on dealing with crisp decisions by passing through gradients for the alternatives of the decisions. There is no smooth transition between the alternatives, which introduces discontinuities in the loss function that hinder learning, which of the alternatives should be chosen. TensorFlow contains a sorting layer (tf.sort) as well as a while loop construct (tf.while_loop). Since the sorting layer only performs a crisp relocation of the gradients and the while loop has a crisp exit condition, there is no gradient with respect to the conditions in these layers. Concurrently, we developed a smooth sorting layer and a smooth while loop. Theoretical work by DeMillo et al. [4] proved that any program could be modeled by a smooth function. Consecutive works [5] - [7] provided approaches for smoothing programs using, i.a., Gaussian smoothing [6] , [7] . We presented AlgoNets as a new kind of layers for neural networks and RANs as a novel technique for solving ill-posed inverse problems. Concurrent with their benefits, AlgoNets, such as the aforementioned rendering layer, can get computationally very expensive. On the other hand, the rendering layer is very powerful since it allows training a 3D reconstruction without 3D supervision using the RAN. Since the RAN is a very complex architecture that requires a very specific training paradigm, it can also take relatively long to train it. To accommodate this issue, we found that by increasing some loss weights and introducing a probability of whether the computation is executed, the training time can be reduced by a factor of two or more. The AlgoNet can also be used in such a way that algorithmic layers solve sub-problems of a given problem to assist a neural network in solving a larger problem. This principle could also be used in the realm of explainable artificial intelligence [13] by adding residual algorithmic layers into neural networks and then analyzing the neurons of the trained AlgoNet. For that, network activation and/or network sensitivity can indicate the relevance of the residual algorithmic layer. To compute the network sensitivity of an algorithmic layer, the gradient with respect to additional weights (constant equal to one) in the algorithmic layer could be computed. By that, similarities between classic algorithms and the behavior of neural networks could be inferred. An alternative approach would be to gradually replace parts of trained neural networks with algorithmic layers and analyzing the effect on the new model accuracy. In the future, we will develop a high-level smooth programming language to improve smooth representations of higher-level programming concepts. Adding trainable weights to the algorithmic layers to improve the accuracy of smooth algorithms and/or allow the rest of the network to influence the behavior of the algorithmic layer is subject to future research. Another future objective is the exploration of neural networks not with a fixed but instead a smooth topology.
Solving inverse problems by using smooth approximations of the forward algorithms to train the inverse models.