economics and game/decision theory [@nisan2007algorithmic], they also provide a highly expressive and transparent language to formulate algorithms in a wide range of domains. In optimization [@bertsimas2011theory] and statistics [@huber2009robust] it has long been observed that competitive optimization is a natural way to encode robustness requirements of algorithms. More recently, researchers in machine learning have been using multi-agent optimization to design highly flexible objective functions for reinforcement learning [@liu2016proximal; @pfau2016connecting; @pathak2017curiosity; @wayne2014hierarchical; @vezhnevets2017feudal] and generative models [@goodfellow2014generative]. We believe that this approach has still a lot of untapped potential, but its full realization depends crucially on the development of efficient and reliable algorithms for the numerical solution of competitive optimization problems. **Gradient descent/ascent and the cycling problem:** For differentiable objective functions, the most naive approach to solving  is gradient descent ascent (GDA), whereby both players independently change their strategy in the direction of steepest descent of their cost function. Unfortunately, this procedure features oscillatory or divergent behavior even in the simple case of a bilinear game ($f(x,y) = x^{\top} y = -g(x,y)$) (see Figure \[fig:bilinear\_strong\]). In game-theoretic terms, GDA lets both players choose their new strategy optimally with respect to the last move of the other player. Thus, the cycling behaviour 