splitting with respect to data items, [@taylor2016training] are able to train on the entire data in aggregate (not sequentially). More recently, [@choromanska2018beyond] have compared the ADMM and BCD variants, also using layer-wise network splitting, but where data are processed individually, more in the style of SGD.

Below is a summary of the ways in which our work differs from, or goes beyond, previous work:

-   No loss functions are used.

-   Networks are split on a finer scale: neurons rather than layers.

-   Our optimizer, RRR, is entirely built from projections.

-   Through warm-start initialization we can train on batches of arbitrary size, up to the entire data set.

-   Variations of the same framework are shown to perform non-negative matrix factorization, classification, and representation learning.

-   The flexibility of the constraint formulation allows us to build a “relabeling classifier" and a non-adversarial generative model.

-   We demonstrate learning on hard, small models where gradient methods fail.

The RRR algorithm, with its projections as computational primitives, has a closer relationship to ADMM than it might seem. First, when using indicator functions for the two objective functions in the ADMM formalism, the minimization steps in ADMM reduce to projections. Moreover, in 