\[alg:gem\] on the MNIST test set. To determine the hyperparameter settings $\gamma$ and $\theta$, we held out a fraction of the training set for validation. Once $\gamma$ and $\theta$ were determined, we trained on the entire training set.

For “deep GEM” we applied GEM to the representation created by GEM, i.e., line 7 of Algorithm \[alg:gem\]. Because of the intermediate nonlinearity this is not equivalent to a single application of GEM, and we do observe an improvement in generalization. Subsequent recursive compositions of GEM degrade generalization, e.g., 3 levels of GEM yields 110 test errors. We would like to better understand the conditions under which composing GEM with itself is beneficial.

Our results occupy an intermediate position amongst state of the art results on MNIST. For comparison we include results from other permutation-invariant methods from [@wan2013regularization] and [@goodfellow2013maxout]. These methods rely on generic non-convex optimization techniques and face challenging scaling issues in a distributed setting [@NIPS2012_0598]. While maximization of the Rayleigh quotient  is non-convex, mature implementations are computationally efficient and numerically robust. The final classifier is built using convex techniques and our pipeline is particularly well suited to the distributed setting, as discussed in Section \[sec:discussion\].

     Method      Test Errors
  ------------- 