for training, all the software for our numerical experiments was implemented with $C$ programs that only call the standard $C$ libraries. Our code runs serially, on a single thread, and without GPU acceleration. Thanks to the “split" nature of the constraints, parallelization could easily have been introduced and indeed it is this feature that partly motivated the related work that we review next.

Relationship to prior work {#sec:ADMM}
==========================

The idea of demoting the pre- and post-activation states of the neurons, from known values determined by forward propagation of the data, to variables that have to be solved along with the network parameters, is not new and has been explored by several groups. Central to this strategy is a scheme for splitting the augmented set of variables into groups amenable to exact, local optimization. For neural network training the two most popular named methods, for acting on the split variables, are alternating direction method of multipliers (ADMM) and block coordinate descent (BCD). In an influential study, [@taylor2016training] used ADMM in networks split along layers, where optimization alternates between the weights of individual layers and the activations that join them, in rough correspondence with, respectively, our constraints and . By also 