autoencoder and classifier, all applications and their implementations by projections are described in complete detail. Although the new approach has the potential to extend the scope of neural networks (e.g. by defining activation not through functions but constraint sets), most of the featured models are standard to allow comparison with stochastic gradient descent.'
author:
- |
    Veit Elser ve10@cornell.edu\
    Department of Physics\
    Cornell University\
    Ithaca, NY 14853-2501, USA
bibliography:
- 'lwl.bib'
title: Learning Without Loss
---

Neural Networks, Training Algorithms, Constraint Satisfaction

Introduction {#sec:intro}
============

When general purpose computers arrived in the 1960s it was realized that certain tasks, such as sorting and Fourier transforms, would be so ubiquitous that it made sense to implement them with provably optimal algorithms. In the present day, as neural networks have become ubiquitous in machine learning systems, the optimality of training algorithms has likewise been the subject of intense research. The expressivity of neural networks makes them attractive for diverse applications but is also the origin of their complexity. Even with the simple piecewise-linear ReLU activation function, the number of linear pieces utilized by a network can in principle grow exponentially with the number of neurons. And while there is choice of loss 