DCEM}(\hat{u}_{1:H}(\cdot;\theta))\vcentcolon=\operatorname{\mathbb{E}}_{x\sim
p(x)}\|u^{\star}_{1:H}(x)-\hat{u}_{1:H}(x;\theta)\|_{2}^{2}.$ (81)

Figure 11 visualizes an example on the cartpole task where this is able to
learn a latent space that captures the cyclic and smoothness structure of the
optimal control sequence space.

Overview. Learning a latent domain with the differentiable cross-entropy
method is a semi-amortization method with ${\mathcal{A}}_{\rm
DCEM}\vcentcolon=(-Q,{\mathcal{U}},{\mathcal{X}},p(x),\pi_{\theta},{\mathcal{L}}_{\rm
reg})$, where the decoder $\varphi_{\theta}$ shows up from the policy
$\pi_{\theta}$ solving the latent optimization problem with DCEM.

####  6.5.9 Iterative amortized policy optimization (IAPO) by Marino et al.
(2020)

IAPO takes a probabilistic view and starts with the observation that DPG and
SVG methods are amortized optimization problems with fully-amortized models
with an objective-based loss. They then suggest to replace the model with an
iterative semi-amortized model where the policy $\pi_{\theta}$ internally
takes gradient steps on the actions of the underlying control optimization
problem, and explore this semi-amortized policy in model-free and model-based
reinforcement learning settings. Thus in the deterministic setting IAPO
performs semi-amortized optimization ${\mathcal{A}}_{\rm
IAPO}\vcentcolon=(-Q,{\mathcal{U}},{\mathcal{X}},p(x),\pi_{\theta},{\mathcal{L}}_{\rm
obj})$.

##  7 Manifold extension: Amortized optimization on the sphere

$f(y;x)$ contours  Optimal $y^{\star}(x)$ Predictions $\hat{y}_{\theta}(x)$
over training

Figure 12: We learn an amortized optimization model to predict the solution to
optimization problems over the sphere. Here we show the predictions throughout
training.

We conclude with a simple new demonstration that applies the insights from
amortized optimization to learn to solve optimization problems over spheres of
the form

$y^{\star}(x)\in\operatorname*{arg\,min}_{y\in{\mathcal{S}}^{2}}f(y;x),$ (82)

where ${\mathcal{S}}^{2}$ is the surface of the _unit 2-sphere_ embedded in
$\mathbb{R}^{3}$ as
${\mathcal{S}}^{2}\vcentcolon=\\{y\in\mathbb{R}^{3}\mid\|y\|_{2}=1\\}$ and $x$
is some parameterization of the function
$f:{\mathcal{S}}^{2}\times{\mathcal{X}}\rightarrow\mathbb{R}$. Eq. 82 is
relevant to physical and geographical settings seeking the extreme values of a
function defined on the Earth or other spaces that can be approximated with a
sphere.

Amortization objective. We first need to turn the constrained problem in eq.
82 into an unconstrained optimization problem of the form eq. 1. In this
setting, one way of doing this is by using a projection:

$y^{\star}(x)\in\operatorname*{arg\,min}_{y\in\mathbb{R}^{3}}f(\pi_{{\mathcal{S}}^{2}}(y);x),$
(83)

where $\pi_{{\mathcal{S}}^{2}}:\mathbb{R}^{3}\rightarrow{\mathcal{S}}^{2}$ is
the Euclidean projection onto ${\mathcal{S}}^{2}$, i.e.,

$\displaystyle\pi_{{\mathcal{S}}^{2}}(x)\vcentcolon=$
$\displaystyle\operatorname*{arg\,min}_{y\in{\mathcal{S}}^{2}}\|y-x\|_{2}$
(84) $\displaystyle=$ $\displaystyle\;x/\|x\|_{2}.$

$c$-convex functions on the sphere. We instantiate a synthetic class of
optimization problems defined on the sphere using the $c$-convex functions
from Cohen et al. (2021):

$f(y;x)={\textstyle\min_{\gamma}}\left\\{\frac{1}{2}d(x,z_{i})+\alpha_{i}\right\\}_{i=1}^{m}$
(85)

where $m$ components define the context $x=\\{z_{i}\\}\cup\\{\alpha_{i}\\}$
with $z_{i}\in{\mathcal{S}}^{2}$ and $\alpha_{i}\in\mathbb{R}$,
$d(x,y)\vcentcolon=\arccos(x^{\top}y)$ is the Riemannian distance on the
sphere in the ambient Euclidean space, and
$\min_{\gamma}(a_{1},\ldots,a_{m})\vcentcolon=-\gamma\log\sum_{i=1}^{m}\exp(-a_{i}/\gamma)$
is a soft minimization operator as proposed in Cuturi and Blondel (2017). We
sample $p(x)$ with $z_{i}\sim{\mathcal{U}}({\mathcal{S}}^{2})$, uniformly from
the sphere, and $\alpha_{i}\sim{\mathcal{N}}(0,\beta)$ with variance
$\beta\in\mathbb{R}_{+}$.

Amortization model. We next instantiate the model
$\hat{y}_{\theta}:{\mathcal{X}}\rightarrow\mathbb{R}$ as a fully-connected
MLP. The predictions to eq. 82 on the sphere can again be obtained by
composing the output with the projection
$\pi_{{\mathcal{S}}^{2}}\circ\hat{y}_{\theta}$.

Optimizing the gradient-based loss. Finally we choose to optimize the
gradient-based loss ${\mathcal{L}}_{\rm obj}$ as the objective and model are
tractable and easily differentiable. Figure 12 shows the model’s predictions
starting with the untrained model and finishing with the trained model,
showing that this setup indeed enables us to predict the solutions to eq. 82
with a single neural network $\hat{y}_{\theta}(x)$ trained with the gradient-
based loss.

Summary. ${\mathcal{A}}_{\rm
sphere}\vcentcolon=(f\circ\pi_{{\mathcal{S}}^{2}},\mathbb{R}^{3},{\mathcal{X}},p(x),\hat{y}_{\theta},{\mathcal{L}}_{\rm
obj})$

##  8 Discussion

Many of the specialized methods discuss tradeoffs and limitations within the
context of their application, and more generally papers such as Chen et al.
(2021a); Metz et al. (2021) provide even deeper probes into general paradigms
for learning to optimize. Here we emphasize a few additional discussion points
around amortized optimization.

###  8.1 Going beyond the known convergence rates of classical methods

Theoretical and empirical optimization research often focuses on discovering
algorithms with theoretically strong convergence rates, and indeed many of the
algorithms with the best convergence rates are used as the state-of-the-art
algorithms in practice. Amortized optimization and the paradigm of learning to
optimize can offer even better convergence rates in this space that surpass
even the best theoretical rates when scoping to optimization problems sampled
from a context space $p(x)$. For example, the fully amortized models for
amortized variational inference and model-free actor-critic methods for RL
solve the optimization problems _in constant time_ with just a single
prediction of the solution from the context without even looking at the
objective! In semi-amortized settings, this intuition also enables
parameterized versions of optimization algorithms to far surpass the known
convergence bounds by similarly giving the amortization model the power of
predicting a near-optimal solution in constant time.

###  8.2 Generalization and convergence guarantees

Despite having powerful successes of amortized optimization in some settings,
the field struggles to bring strong success in other domains. Despite having
the capacity of surpassing the convergence rates of other algorithms,
oftentimes in practice amortized optimization methods can deeply struggle to
generalize and converge to reasonable solutions. In some deployments this
inaccuracy may be acceptable if there is a quick way of checking the quality
of the amortized model, e.g. the residuals for fixed-point and convex
problems. If that is the case, then poorly-solved instances can be flagged and
re-solved with a standard solver for the problem that may incur more
computational time for that instance.

###  8.3 Measuring performance

Quantifying the performance of amortization models can even more challenging
than the choice between using a regression- or objective-based loss and is
often tied to problem-specific metrics that are important. For example, even
if a method is able to attain low objective values in a few iterations, the
computation may take _longer_ than a specialized algorithm or another
amortization model that can reach the same level of accuracy, thus not making
it useful for the original goal of speeding up solves to eq. 1.

###  8.4 Successes and limitations of amortized optimization

While amortized optimization has standout applications in variational
inference, reinforcement learning, and meta-learning, it struggles to bring
value in other settings. Often, learning the amortized model is
computationally more expensive than solving the original optimization problems
and brings instabilities into a higher-level learning or optimization process
deployed on top of potentially inaccurate solutions from the amortized model.
Here we summarize principles behind successful applications of amortized
optimization and characterize limitations that may arise.

####  8.4.1 Characteristics of successful applications

  * •

Objective $f(y;x)$ is smooth over the domain ${\mathcal{Y}}$ and has unique
solutions $y^{\star}$. With objective-based learning, non-convex objectives
with few poor local optimum are ideal. This behavior can be encouraged with
smoothing as is often done for meta-learning and policy learning (section
4.3.2).

  * •

A higher-level process should tolerate sub-optimal solutions given by
$\hat{y}$ in the beginning of training. In variational encoders, the
suboptimal bound on the likelihood is still acceptable to optimize the density
model’s parameters over in eq. 35. And in reinforcement learning policies, a
suboptimal solution to the maximum value problem is still acceptable to deploy
on the system in early phases of training, and may even be desirable for the
exploration induced by randomly initialized policies.

  * •

The context distribution $p(x)$ is not too big and well-scoped and deployed on
a specialized class of sub-problems. For example, instead of trying to
amortize the solution to _every_ possible $\operatorname{ELBO}$ maximization,
VAEs amortize the problem only over the dataset the density model is being
trained on. And in reinforcement learning, the policy $\pi_{\theta}$ doesn’t
try to amortize the solution to _every_ possible control problem, but instead
focuses only on amortizing the solutions to the control problems on the replay
buffer of the specific MDP.

  * •

In semi-amortized models, parameterizing the initialization and specialized
components for the updates. While semi-amortized models are a thriving
research topic, the most successful applications of them:

    1. 1.

Parameterize and learn the initial iterate. MAML (Finn et al., 2017) _only_
parameterizes the initial iterate and follows it with gradient descent steps.
Bai et al. (2022) parameterizes the initial iterate and follows it with
accelerated fixed-point iterations.

    2. 2.

Parameterize and learn specialized components of the updates. In sparse
coding, LISTA (Gregor and LeCun, 2010) only parameterized $\\{F,G,\beta\\}$
instead of the entire update rule. Bai et al. (2022) only parameterizes
$\alpha,\beta$ after the initial iterate, and RLQP (Ichnowski et al., 2021)
only parameterizing $\rho$.

While using a pure sequence model to update a sequence of iterations is
possible and theoretically satisfying as it gives the model the power to
arbitrarily update the sequence of iterates, in practice this can be unstable
and severely overfit to the training instances. Metz et al. (2021) observes,
for example, that semi-amortized recurrent sequence models induce chaotic
behaviors and exploding gradients.

####  8.4.2 Limitations and failures

  * •

Amortized optimization does _not_ magically solve otherwise intractable
optimization problems! At least not without significant insights. In most
successful settings, the original optimization problem can be (semi-)tractably
solved for a context $x$ with classical methods, such as using standard black-
box variational inference or model-predictive control methods.
Intractabilities indeed start arising when repeatedly solving the optimization
problem, even if a single one can be reasonably solved, and amortization often
thrive in these settings to rapidly solve problems with similar structure.

  * •

The combination of $p(x)$ and $y^{\star}(x)$ are too hard for a model to
learn. This could come from $p(x)$ being too large, e.g. contexts of every
optimization problem in the universe, or the solution $y^{\star}(x)$ not being
smooth or predictable. $y^{\star}(x)$ may also not be unique, but this is
perhaps easier to handle if the loss is carefully set up, e.g. objective-based
losses handle this more nicely.

  * •

The domain requires accurate solutions. Even though metrics that measure the
solution quality of $\hat{y}$ can be defined on top of eq. 1, amortized
methods typically cannot rival the accuracy of standard algorithms used to
solve the optimization problems. In these settings, amortized optimization
still has the potential at uncovering new foundations and algorithms for
solving problems, but is non-trivial to successfully demonstrate. From an
amortization perspective, one difficulty of safety-critical model-free
reinforcement learning comes from needing to ensure the amortized policy
properly optimizes a value estimate that (hopefully) encodes safety-critical
properties of the state-action space.

###  8.5 Some open problems and under-explored directions

In most domains, introducing or significantly improving amortized optimization
is extremely valuable and will likely be well-received. Beyond this, there are
many under-explored directions and combinations of ideas covered in this
tutorial that can be shared between the existing fields using amortized
optimization, for example:

  1. 1.

Overcoming local minima with objective-based losses and connections to
stochastic policies. In section 4.3.2, we covered the objective smoothing by
Metz et al. (2019a); Merchant et al. (2021) to overcome local minima in the
objective. These have striking similarities to stochastic policies in
reinforcement learning that also overcome local minima, e.g. in eq. 64. The
stochastic policies, such as in Haarnoja et al. (2018), have the desirable
property of starting with a high variance and then focusing in on a low-
variance solution with a penalty constraining the entropy to a fixed value. I
have not seen this property in many amortization settings and it seems
desirable for the same reasons.

  2. 2.

Widespread and usable amortized convex solvers. When using off-the-shelf
optimization packages such as Diamond and Boyd (2016); O’donoghue et al.
(2016); Stellato et al. (2018), users are likely solving many similar problem
instances that amortization can help improve. Venkataraman and Amos (2021);
Ichnowski et al. (2021) are active research directions that study adding
amortization to these solvers, but they do not scale to the general online
setting that also doesn’t add too much learning overhead for the user.

  3. 3.

Improving the wall-clock training time of implicit models and differentiable
optimization. Optimization problems and fixed-point problems are being
integrated into machine learning models, such as with differentiable
optimization (Domke, 2012; Gould et al., 2016; Amos and Kolter, 2017; Amos,
2019; Agrawal et al., 2019a; Lee et al., 2019b) and deep equilibrium models
(Bai et al., 2019, 2020). In these settings, the data distribution the model
is being trained on naturally induces a distribution over contexts that seem
amenable to amortization. Venkataraman and Amos (2021); Bai et al. (2022)
explore amortization in these settings, but often do not improve the wall-
clock time it takes to train these models from scratch.

  4. 4.

Understanding the amortization gap. Cremer et al. (2018) study the
_amortization gap_ in amortized variational inference, which measures how well
the amortization model approximates the true solution. This crucial concept
should be analyzed in most amortized optimization settings to understand the
accuracy of the amortization model.

  5. 5.

Implicit differentiation and shrinkage. Chen et al. (2019); Rajeswaran et al.
(2019) show that penalizing the amortization objective can significantly
improve the computational and memory requirements to train a semi-amortized
model for meta-learning. Many of the ideas in these settings can be applied in
other amortization settings, as also observed by Huszár (2019).

  6. 6.

Amortized and semi-amortized control and reinforcement learning. Applications
of semi-amortization in control and reinforcement learning that we covered in
section 6.5 are budding and learning sample-efficient optimal controllers is
an active research area, especially in model-based settings where the dynamics
model is known or approximated. Amos and Yarats (2019) shows how amortization
can learn latent control spaces that are aware of the structure of the
solutions to control problems. Marino et al. (2020) study semi-amortized
methods based on gradient descent and show that they better-amortize the
solutions than the standard fully-amortized models.

###  8.6 Coding and implementing amortized optimization

The approaches we consider here are most-easily implemented with automatic
differentiation software such as Maclaurin et al. (2015b); Al-Rfou et al.
(2016); Abadi et al. (2016); Bezanson et al. (2017); Agrawal et al. (2019b);
Paszke et al. (2019); Bradbury et al. (2020). Many of the approaches cited in
here have open-source implementations using many of these packages which
provide a good starting point to start building on top of.

Implementing semi-amortized models are usually more challenging than fully-
amortized models. Learning an optimization-based model that internally solves
an optimization problem is not as widespread as learning a feedforward neural
network. While most autodiff packages provide standalone features to implement
unrolled gradient-based optimization, the following specialized packages
provide crucial features that further enable the exploration of semi-amortized
models:

  * •

cvxpylayers (Agrawal et al., 2019a) allows an optimization problem to be
expressed in the high-level language `CVXPY` (Diamond and Boyd, 2016) and
exported to PyTorch, JAX, and TensorFlow as a differentiable optimization
layers.

  * •

jaxopt (Blondel et al., 2021) is a differentiable optimization library for JAX
and implements many optimization settings and fixed-point computations along
with their implicit derivatives.

  * •

higher (Grefenstette et al., 2019) is a PyTorch library that adds
differentiable higher-order optimization support with 1) monkey-patched
functional `torch.nn` modules, and 2) differentiable versions of `torch.optim`
optimizers such as Adam and SGD. This enables arbitrary torch modules and
optimizers to be unrolled and used as as a semi-amortized model.

  * •

DiffOpt.jl provides differentiable optimization in Julia’s JuMP (Dunning et
al., 2017).

  * •

Torchmeta (Deleu et al., 2019) and learn2learn (Arnold et al., 2020) are
PyTorch libraries and collection of meta-learning algorithms that also focus
on making data-loading and task definitions easy.

##  9 Related work

###  9.1 Other tutorials, reviews, and discussions on amortized optimization

My goal in writing this tutorial was to provide a perspective of existing
amortized optimization methods for learning to optimize with a categorization
of the modeling (fully-amortized and semi-amortized) and learning (gradient-
based, objective-based, or RL-based) aspects that I have found useful and have
not seen emphasized as much in the literature. The other tutorials and reviews
on amortized optimization, learning to optimize, and meta-learning over
continuous domains that I am aware of are excellent resources:

  * •

Chen et al. (2021a) captures many other emerging areas of learning to optimize
and discuss many other modeling paradigms and optimization methods for
learning to optimize, such as plug-and-play methods (Venkatakrishnan et al.,
2013; Meinhardt et al., 2017; Rick Chang et al., 2017; Zhang et al., 2017).
They emphasize the key aspects and questions to tackle as a community,
including model capacity, trainability, generalization, and interpretability.
They propose _Open-L2O_ as a new benchmark for learning to optimize and review
many other applications, including sparse and low-rank regression, graphical
models, differential equations, quadratic optimization, inverse problems,
constrained optimization, image restoration and reconstruction, medical and
biological imaging, wireless communications, seismic imaging.

  * •

Shu (2017) is a blog post that discusses fully-amortized models with gradient-
based learning and includes applications in variational inference, meta-
learning, image style transfer, and survival-based classification.

  * •

Weng (2018) is a blog post with an introduction and review of meta-learning
methods. After defining the problem setup, the review discusses metric-based,
model-based, and optimization-based approaches, and discusses approximations
to the second-order derivatives that come up with MAML.

  * •

Arridge et al. (2019) survey data-driven optimization methods for solving
inverse problems. This survey covers the foundations and discuss tradeoffs
between statistical accuracy and robustness, and functional complexity,
stability, and interpretability. Their applications include image denoising,
image deblurring, image inpainting, computed tomography, photoacoustic
tomography, magnetic resonance imaging, and magnetic particle imaging.

  * •

Hospedales et al. (2020) is a review focused on meta-learning, where they
categorize meta-learning components into a meta-representation, meta-
optimizer, and meta-objective. The most relevant connections to amortization
here are that the meta-representation can instantiate an amortized
optimization problem that is solved with the meta-optimizer.

  * •

Kim (2020) is a dissertation on deep latent variable models for natural
language and contextualizes and studies the use of amortization and semi-
amortization in this setting.

  * •

Banert et al. (2020) consider theoretical foundations for data-driven
nonsmooth optimization and show applications in deblurring and solving inverse
problems for computed tomography and is further developed in Banert et al.
(2021).

  * •

Marino (2021) is a dissertation on learned feedback and feedforward
information for perception and control and contextualizes and studies the use
of amortization and semi-amortization in these settings.

  * •

Monga et al. (2021) is a review on algorithm unrolling that starts with the
unrolling in LISTA (Gregor and LeCun, 2010) for amortized sparse coding, and
then connects to other methods of unrolling specialized algorithms. While some
unrolling methods have applications in semi-amortized models, this review also
considers applications and use-cases beyond just amortized optimization.

  * •

Liu et al. (2022) study fully-amortized models based on deep sets (Zaheer et
al., 2017) and set transformers (Lee et al., 2019a). They consider regression-
and objective-based losses for regression, PCA, core-set creation, and supply
management for cyber-physical systems.

###  9.2 Amortized optimization over discrete domains

A significant generalization of eq. 1 is to optimization problems that have
discrete domains, which includes combinatorial optimization and mixed
discrete-continuous optimization. I have chosen to not include these works in
this tutorial as many methods for discrete optimization are significantly
different from what we cover here, as learning with derivative information
often becomes impossible. Key works in discrete and combinatorial spaces
include Khalil et al. (2016); Dai et al. (2017); Jeong and Song (2019);
Bertsimas and Stellato (2019); Shao et al. (2021); Bertsimas and Stellato
(2021); Cappart et al. (2021) and the surveys (Lodi and Zarpellon, 2017;
Bengio et al., 2021; Kotary et al., 2021) capture a much broader view of this
space. Banerjee and Roy (2015) consider repeated ILP solves and show
applications in aircraft carrier deck scheduling and vehicle routing. For
architecture search, Luo et al. (2018) learn a continuous latent space behind
the discrete architecture space. Many reinforcement learning and control
methods over discrete spaces can also be seen as amortizing or semi-amortizing
the discrete control problems, for example: Cauligi et al. (2020, 2021) use
regression-based amortization to learn mixed-integer control policies.
Fickinger et al. (2021) fine-tune the policy optimizer for every encountered
state. Tennenholtz and Mannor (2019); Chandak et al. (2019); Van de Wiele et
al. (2020) learn latent action spaces for high-dimensional discrete action
spaces with shared structure.

###  9.3 Learning-augmented and amortized algorithms beyond optimization

While many algorithms can be interpreted as solving an optimization problems
or fixed-point computations and can therefore be improved with amortized
optimization, it is also fruitful to use learning to improve algorithms that
have nothing to do with optimization. Some key starting references in this
space include data-driven algorithm design (Balcan, 2020), learning to prune
(Alabi et al., 2019), learning solutions to differential equations (Li et al.,
2020; Poli et al., 2020; Karniadakis et al., 2021; Kovachki et al., 2021; Chen
et al., 2021b; Blechschmidt and Ernst, 2021; Marwah et al., 2021; Berto et
al., 2021) learning simulators for physics (Grzeszczuk et al., 1998; Ladickỳ
et al., 2015; He et al., 2019; Sanchez-Gonzalez et al., 2020; Wiewel et al.,
2019; Usman et al., 2021; Vinuesa and Brunton, 2021), and learning for
symbolic math (Lample and Charton, 2019; Charton, 2021; Charton et al., 2021;
Drori et al., 2021; d’Ascoli et al., 2022) Salimans and Ho (2022)
progressively amortizes a sampling process for diffusion models.

###  9.4 Continuation and homotopy methods

Amortized optimization settings share a similar motivation to continuation and
homotopy methods that have been studied for over four decades (Richter and
Decarlo, 1983; Watson and Haftka, 1989; Allgower and Georg, 2012). These
methods usually set the context space to be the interval ${\mathcal{X}}=[0,1]$
and simultaneously solve (without learning) problems along this line. This
similarity indicates that problem classes typically studied by continuation
and homotopy methods could also benefit from the shared amortization models
here.

Acknowledgments

I would like to thank Nil-Jana Akpinar, Samuel Cohen, Georgina Hall, Boris
Knyazev, Hane Lee, Joe Marino, Maximilian Nickel, Rajiv Sambharya, Bartolomeo
Stellato, Alex Terenin, Eugene Vinitsky, and Atlas Wang for insightful
discussions and feedback on this tutorial. Additional thoughts, insights,
corrections, and references are very welcome<EMAIL_ADDRESS>
## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: A system for large-scale machine learning.  In _12th USENIX symposium on operating systems design and implementation (OSDI 16)_ , pages 265–283, 2016.
  * Absil et al. (2009) P-A Absil, Robert Mahony, and Rodolphe Sepulchre.  _Optimization algorithms on matrix manifolds_.  Princeton University Press, 2009.
  * Adams and Zemel (2011) Ryan Prescott Adams and Richard S Zemel.  Ranking via sinkhorn propagation.  _arXiv preprint arXiv:1106.1925_ , 2011.
  * Agrawal et al. (2019a) Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter.  Differentiable convex optimization layers.  In _Advances in Neural Information Processing Systems_ , pages 9558–9570, 2019a.
  * Agrawal et al. (2019b) Akshay Agrawal, Akshay Naresh Modi, Alexandre Passos, Allen Lavoie, Ashish Agarwal, Asim Shankar, Igor Ganichev, Josh Levenberg, Mingsheng Hong, Rajat Monga, et al.  Tensorflow eager: A multi-stage, python-embedded dsl for machine learning.  _arXiv preprint arXiv:1903.01855_ , 2019b.
  * Aho et al. (1986) Alfred V Aho, Ravi Sethi, and Jeffrey D Ullman.  Compilers, principles, techniques.  _Addison wesley_ , 7(8):9, 1986.
  * Al-Rfou et al. (2016) Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, Alexander Belopolsky, et al.  Theano: A python framework for fast computation of mathematical expressions.  _arXiv e-prints_ , pages arXiv–1605, 2016.
  * Alabi et al. (2019) Daniel Alabi, Adam Tauman Kalai, Katrina Liggett, Cameron Musco, Christos Tzamos, and Ellen Vitercik.  Learning to prune: Speeding up repeated computations.  In _Conference on Learning Theory_ , pages 30–33. PMLR, 2019.
  * Ali et al. (2017) Alnur Ali, Eric Wong, and J Zico Kolter.  A semismooth newton method for fast, generic convex programming.  In _International Conference on Machine Learning_ , pages 70–79. PMLR, 2017.
  * Allgower and Georg (2012) Eugene L Allgower and Kurt Georg.  _Numerical continuation methods: an introduction_ , volume 13.  Springer Science & Business Media, 2012.
  * Amos (2019) Brandon Amos.  _Differentiable Optimization-Based Modeling for Machine Learning_.  PhD thesis, Carnegie Mellon University, May 2019.
  * Amos and Kolter (2017) Brandon Amos and J Zico Kolter.  Optnet: Differentiable optimization as a layer in neural networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 136–145. JMLR. org, 2017.
  * Amos and Yarats (2019) Brandon Amos and Denis Yarats.  The differentiable cross-entropy method.  _arXiv preprint arXiv:1909.12830_ , 2019.
  * Amos et al. (2017) Brandon Amos, Lei Xu, and J Zico Kolter.  Input convex neural networks.  In _International Conference on Machine Learning_ , pages 146–155. PMLR, 2017.
  * Amos et al. (2018) Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter.  Differentiable mpc for end-to-end planning and control.  In _Advances in Neural Information Processing Systems_ , pages 8289–8300, 2018.
  * Amos et al. (2019) Brandon Amos, Vladlen Koltun, and J Zico Kolter.  The limited multi-label projection layer.  _arXiv preprint arXiv:1906.08707_ , 2019.
  * Amos et al. (2021) Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson.  On the model-based stochastic value gradient for continuous reinforcement learning.  In _Learning for Dynamics and Control_ , pages 6–20. PMLR, 2021.
  * Anderson (1965) Donald G Anderson.  Iterative procedures for nonlinear integral equations.  _Journal of the ACM (JACM)_ , 12(4):547–560, 1965\.
  * Andrychowicz et al. (2016) Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  In _Advances in neural information processing systems_ , pages 3981–3989, 2016.
  * Antoniou et al. (2018) Antreas Antoniou, Harrison Edwards, and Amos Storkey.  How to train your maml.  _arXiv preprint arXiv:1810.09502_ , 2018.
  * Arbel and Mairal (2021) Michael Arbel and Julien Mairal.  Amortized implicit differentiation for stochastic bilevel optimization.  _arXiv preprint arXiv:2111.14580_ , 2021.
  * Arnold et al. (2020) Sébastien MR Arnold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias.  learn2learn: A library for meta-learning research.  _arXiv preprint arXiv:2008.12284_ , 2020.
  * Arridge et al. (2019) Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb.  Solving inverse problems using data-driven models.  _Acta Numerica_ , 28:1–174, 2019.
  * Bai et al. (2019) Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  Deep equilibrium models.  _arXiv preprint arXiv:1909.01377_ , 2019.
  * Bai et al. (2020) Shaojie Bai, Vladlen Koltun, and J Zico Kolter.  Multiscale deep equilibrium models.  _arXiv preprint arXiv:2006.08656_ , 2020.
  * Bai et al. (2022) Shaojie Bai, Vladlen Koltun, and J Zico Kolter.  Neural deep equilibrium solvers.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=B0oHOwT5ENL.
  * Baker (2019) Kyri Baker.  Learning warm-start points for ac optimal power flow.  In _2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)_ , pages 1–6. IEEE, 2019.
  * Balcan (2020) Maria-Florina Balcan.  Data-driven algorithm design.  _arXiv preprint arXiv:2011.07177_ , 2020.
  * Banerjee and Roy (2015) Ashis Gopal Banerjee and Nicholas Roy.  Efficiently solving repeated integer linear programming problems by learning solutions of similar linear programming problems using boosting trees.  _MIT_ , 2015.
  * Banert et al. (2020) Sebastian Banert, Axel Ringh, Jonas Adler, Johan Karlsson, and Ozan Oktem.  Data-driven nonsmooth optimization.  _SIAM Journal on Optimization_ , 30(1):102–131, 2020.
  * Banert et al. (2021) Sebastian Banert, Jevgenija Rudzusika, Ozan Öktem, and Jonas Adler.  Accelerated forward-backward optimization using deep learning.  _arXiv preprint arXiv:2105.05210_ , 2021.
  * Bank et al. (1982) Bernd Bank, Jürgen Guddat, Diethard Klatte, Bernd Kummer, and Klaus Tammer.  _Non-linear parametric optimization_.  Springer, 1982.
  * Barratt (2018) Shane Barratt.  On the differentiability of the solution to convex optimization problems.  _arXiv preprint arXiv:1804.05098_ , 2018.
  * Baxter (1998) Jonathan Baxter.  Theoretical models of learning to learn.  In _Learning to learn_ , pages 71–94. Springer, 1998.
  * Beck and Teboulle (2009) Amir Beck and Marc Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  _SIAM journal on imaging sciences_ , 2(1):183–202, 2009.
  * Belanger (2017) David Belanger.  _Deep energy-based models for structured prediction_.  PhD thesis, University of Massachusetts Amherst, 2017.
  * Belanger and McCallum (2016) David Belanger and Andrew McCallum.  Structured prediction energy networks.  In _International Conference on Machine Learning_ , pages 983–992, 2016.
  * Belanger et al. (2017) David Belanger, Bishan Yang, and Andrew McCallum.  End-to-end learning for structured prediction energy networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 429–439. JMLR. org, 2017.
  * Bello et al. (2017) Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le.  Neural optimizer search with reinforcement learning.  In _International Conference on Machine Learning_ , pages 459–468. PMLR, 2017.
  * Bengio et al. (1994) Samy Bengio, Yoshua Bengio, and Jocelyn Cloutier.  Use of genetic programming for the search of a new learning rule for neural networks.  In _Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence_ , pages 324–327. IEEE, 1994.
  * Bengio et al. (2021) Yoshua Bengio, Andrea Lodi, and Antoine Prouvost.  Machine learning for combinatorial optimization: a methodological tour d’horizon.  _European Journal of Operational Research_ , 290(2):405–421, 2021.
  * Bertinetto et al. (2018) Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi.  Meta-learning with differentiable closed-form solvers.  _arXiv preprint arXiv:1805.08136_ , 2018.
  * Berto et al. (2021) Federico Berto, Stefano Massaroli, Michael Poli, and Jinkyoo Park.  Neural solvers for fast and accurate numerical optimal control.  In _The Symbiosis of Deep Learning and Differential Equations_ , 2021\.
  * Bertsekas (1971) Dimitri P Bertsekas.  _Control of uncertain systems with a set-membership description of the uncertainty._ PhD thesis, Massachusetts Institute of Technology, 1971.
  * Bertsimas and Stellato (2019) Dimitris Bertsimas and Bartolomeo Stellato.  Online mixed-integer optimization in milliseconds.  _arXiv preprint arXiv:1907.02206_ , 2019.
  * Bertsimas and Stellato (2021) Dimitris Bertsimas and Bartolomeo Stellato.  The voice of optimization.  _Machine Learning_ , 110(2):249–277, 2021.
  * Bezanson et al. (2017) Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah.  Julia: A fresh approach to numerical computing.  _SIAM review_ , 59(1):65–98, 2017.
  * Bhardwaj et al. (2020) Mohak Bhardwaj, Byron Boots, and Mustafa Mukadam.  Differentiable gaussian process motion planning.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 10598–10604. IEEE, 2020.
  * Blechschmidt and Ernst (2021) Jan Blechschmidt and Oliver G Ernst.  Three ways to solve partial differential equations with neural networks—a review.  _GAMM-Mitteilungen_ , page e202100006, 2021.
  * Blei et al. (2017) David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518):859–877, 2017.
  * Blondel (2019) Mathieu Blondel.  Structured prediction with projection oracles.  _arXiv preprint arXiv:1910.11369_ , 2019.
  * Blondel et al. (2020) Mathieu Blondel, André FT Martins, and Vlad Niculae.  Learning with fenchel-young losses.  _J. Mach. Learn. Res._ , 21(35):1–69, 2020.
  * Blondel et al. (2021) Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert.  Efficient and modular implicit differentiation.  _arXiv preprint arXiv:2105.15183_ , 2021.
  * Bonnans and Shapiro (2013) J Frédéric Bonnans and Alexander Shapiro.  _Perturbation analysis of optimization problems_.  Springer Science & Business Media, 2013.
  * Boyd et al. (2011) Stephen Boyd, Neal Parikh, and Eric Chu.  _Distributed optimization and statistical learning via the alternating direction method of multipliers_.  Now Publishers Inc, 2011.
  * Bradbury et al. (2020) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne.  Jax: composable transformations of python+ numpy programs, 2018.  _URL http://github. com/google/jax_ , 4:16, 2020.
  * Broyden (1965) Charles G Broyden.  A class of methods for solving nonlinear simultaneous equations.  _Mathematics of computation_ , 19(92):577–593, 1965.
  * Busseti et al. (2019) Enzo Busseti, Walaa M Moursi, and Stephen Boyd.  Solution refinement at regular points of conic problems.  _Computational Optimization and Applications_ , 74(3):627–643, 2019.
  * Byravan et al. (2019) Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller.  Imagined value gradients: Model-based policy optimization with transferable latent dynamics models.  _arXiv preprint arXiv:1910.04142_ , 2019.
  * Byravan et al. (2021) Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim, Mehdi Mirza, Alessandro Davide Ialongo, Yuval Tassa, Jost Tobias Springenberg, Abbas Abdolmaleki, Nicolas Heess, Josh Merel, et al.  Evaluating model-based planning and planner amortization for continuous control.  _arXiv preprint arXiv:2110.03363_ , 2021.
  * Cappart et al. (2021) Quentin Cappart, Didier Chételat, Elias Khalil, Andrea Lodi, Christopher Morris, and Petar Veličković.  Combinatorial optimization and reasoning with graph neural networks.  _arXiv preprint arXiv:2102.09544_ , 2021.
  * Carter (2001) Michael Carter.  _Foundations of mathematical economics_.  MIT press, 2001.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Cauligi et al. (2020) Abhishek Cauligi, Preston Culbertson, Bartolomeo Stellato, Dimitris Bertsimas, Mac Schwager, and Marco Pavone.  Learning mixed-integer convex optimization strategies for robot planning and control.  In _2020 59th IEEE Conference on Decision and Control (CDC)_ , pages 1698–1705. IEEE, 2020.
  * Cauligi et al. (2021) Abhishek Cauligi, Preston Culbertson, Edward Schmerling, Mac Schwager, Bartolomeo Stellato, and Marco Pavone.  Coco: Online mixed-integer control via supervised learning.  _IEEE Robotics and Automation Letters_ , 2021.
  * Chandak et al. (2019) Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas.  Learning action representations for reinforcement learning.  In _International Conference on Machine Learning_ , pages 941–950. PMLR, 2019.
  * Charton (2021) François Charton.  Linear algebra with transformers.  _arXiv preprint arXiv:2112.01898_ , 2021.
  * Charton et al. (2021) François Charton, Amaury Hayat, Sean T McQuade, Nathaniel J Merrill, and Benedetto Piccoli.  A deep language model to predict metabolic network equilibria.  _arXiv preprint arXiv:2112.03588_ , 2021.
  * Chen et al. (2001) Scott Shaobing Chen, David L Donoho, and Michael A Saunders.  Atomic decomposition by basis pursuit.  _SIAM review_ , 43(1):129–159, 2001.
  * Chen et al. (2022) Steven W Chen, Tianyu Wang, Nikolay Atanasov, Vijay Kumar, and Manfred Morari.  Large scale model predictive control with neural networks and primal active sets.  _Automatica_ , 135:109947, 2022.
  * Chen et al. (2021a) Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin.  Learning to optimize: A primer and a benchmark.  _arXiv preprint arXiv:2103.12828_ , 2021a.
  * Chen et al. (2021b) Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart.  Solving and learning nonlinear pdes with gaussian processes.  _arXiv preprint arXiv:2103.12959_ , 2021b.
  * Chen et al. (2017) Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando Freitas.  Learning to learn without gradient descent by gradient descent.  In _International Conference on Machine Learning_ , pages 748–756. PMLR, 2017.
  * Chen et al. (2019) Yutian Chen, Abram L Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew W Hoffman, and Nando de Freitas.  Modular meta-learning with shrinkage.  _arXiv preprint arXiv:1909.05557_ , 2019.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Cohen et al. (2021) Samuel Cohen, Brandon Amos, and Yaron Lipman.  Riemannian convex potential maps.  _arXiv preprint arXiv:2106.10272_ , 2021.
  * Cover and Thomas (2006) Thomas M Cover and Joy A Thomas.  Elements of information theory (wiley series in telecommunications and signal processing), 2006.
  * Cremer et al. (2018) Chris Cremer, Xuechen Li, and David Duvenaud.  Inference suboptimality in variational autoencoders.  _arXiv preprint arXiv:1801.03558_ , 2018.
  * Cuturi and Blondel (2017) Marco Cuturi and Mathieu Blondel.  Soft-dtw: a differentiable loss function for time-series.  In _International Conference on Machine Learning_ , pages 894–903. PMLR, 2017.
  * Dai et al. (2017) Hanjun Dai, Elias B Khalil, Yuyu Zhang, Bistra Dilkina, and Le Song.  Learning combinatorial optimization algorithms over graphs.  _arXiv preprint arXiv:1704.01665_ , 2017.
  * Danskin (1966) John M Danskin.  The theory of max-min, with applications.  _SIAM Journal on Applied Mathematics_ , 14(4):641–664, 1966.
  * d’Ascoli et al. (2022) Stéphane d’Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and François Charton.  Deep symbolic regression for recurrent sequences, 2022.
  * Daubechies et al. (2004) Ingrid Daubechies, Michel Defrise, and Christine De Mol.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.  _Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences_ , 57(11):1413–1457, 2004.
  * Davidson and Jinturkar (1995) Jack W Davidson and Sanjay Jinturkar.  An aggressive approach to loop unrolling.  Technical report, Citeseer, 1995.
  * Dayan et al. (1995) Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel.  The helmholtz machine.  _Neural computation_ , 7(5):889–904, 1995.
  * De Boer et al. (2005) Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein.  A tutorial on the cross-entropy method.  _Annals of operations research_ , 134(1):19–67, 2005.
  * Deisenroth and Rasmussen (2011) Marc Deisenroth and Carl E Rasmussen.  Pilco: A model-based and data-efficient approach to policy search.  In _Proceedings of the 28th International Conference on machine learning (ICML-11)_ , pages 465–472, 2011.
  * Deleu et al. (2019) Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio.  Torchmeta: A meta-learning library for pytorch.  _arXiv preprint arXiv:1909.06576_ , 2019.
  * Diamond and Boyd (2016) Steven Diamond and Stephen Boyd.  Cvxpy: A python-embedded modeling language for convex optimization.  _The Journal of Machine Learning Research_ , 17(1):2909–2913, 2016.
  * Dini (1878) Ulisse Dini.  _Analisi infinitesimale_.  Lithografia Gorani, 1878.
  * Doersch (2016) Carl Doersch.  Tutorial on variational autoencoders.  _arXiv preprint arXiv:1606.05908_ , 2016.
  * Domke (2012) Justin Domke.  Generic methods for optimization-based modeling.  In _Artificial Intelligence and Statistics_ , pages 318–326. PMLR, 2012.
  * Dong et al. (2020) Wenqian Dong, Zhen Xie, Gokcen Kestor, and Dong Li.  Smart-pgsim: using neural network to accelerate ac-opf power grid simulation.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–15. IEEE, 2020.
  * Donoho and Elad (2003) David L Donoho and Michael Elad.  Optimally sparse representation in general (nonorthogonal) dictionaries via $\ell$1 minimization.  _Proceedings of the National Academy of Sciences_ , 100(5):2197–2202, 2003.
  * Dontchev and Rockafellar (2009) Asen L Dontchev and R Tyrrell Rockafellar.  _Implicit functions and solution mappings_ , volume 543.  Springer, 2009.
  * Donti et al. (2021) Priya L Donti, David Rolnick, and J Zico Kolter.  Dc3: A learning method for optimization with hard constraints.  _arXiv preprint arXiv:2104.12225_ , 2021.
  * Drori et al. (2021) Iddo Drori, Sunny Tran, Roman Wang, Newman Cheng, Kevin Liu, Leonard Tang, Elizabeth Ke, Nikhil Singh, Taylor L. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang.  A neural network solves and generates mathematics problems by program synthesis: Calculus, differential equations, linear algebra, and more, 2021.
  * Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  _Journal of machine learning research_ , 12(7), 2011.
  * Dunning et al. (2017) Iain Dunning, Joey Huchette, and Miles Lubin.  Jump: A modeling language for mathematical optimization.  _SIAM Review_ , 59(2):295–320, 2017.  doi: 10.1137/15M1020575.
  * Duruisseaux and Leok (2022) Valentin Duruisseaux and Melvin Leok.  Accelerated optimization on riemannian manifolds via projected variational integrators, 2022.
  * Fiacco (2020) Anthony V Fiacco.  _Mathematical programming with data perturbations_.  CRC Press, 2020.
  * Fiacco and Ishizuka (1990) Anthony V Fiacco and Yo Ishizuka.  Sensitivity and stability analysis for nonlinear programming.  _Annals of Operations Research_ , 27(1):215–235, 1990.
  * Fickinger et al. (2021) Arnaud Fickinger, Hengyuan Hu, Brandon Amos, Stuart Russell, and Noam Brown.  Scalable online planning via reinforcement learning fine-tuning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1126–1135. JMLR. org, 2017.
  * Foerster et al. (2017) Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch.  Learning with opponent-learning awareness.  _arXiv preprint arXiv:1709.04326_ , 2017.
  * Franceschi et al. (2018) Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil.  Bilevel programming for hyperparameter optimization and meta-learning.  In _International Conference on Machine Learning_ , pages 1568–1577. PMLR, 2018.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke van Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , 2018.
  * Gao et al. (2020) Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi.  Learning to optimize on spd manifolds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7700–7709, 2020.
  * Gershman and Goodman (2014) Samuel Gershman and Noah Goodman.  Amortized inference in probabilistic reasoning.  In _Proceedings of the annual meeting of the cognitive science society_ , volume 36, 2014.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Gordon et al. (2018) Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner.  Meta-learning probabilistic inference for prediction.  _arXiv preprint arXiv:1805.09921_ , 2018.
  * Gould et al. (2016) Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo.  On differentiating parameterized argmin and argmax problems with application to bi-level optimization.  _arXiv preprint arXiv:1607.05447_ , 2016.
  * Grefenstette et al. (2019) Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala.  Generalized inner loop meta-learning.  _arXiv preprint arXiv:1910.01727_ , 2019.
  * Gregor and LeCun (2010) Karol Gregor and Yann LeCun.  Learning fast approximations of sparse coding.  In _Proceedings of the 27th international conference on international conference on machine learning_ , pages 399–406, 2010.
  * Grzeszczuk et al. (1998) Radek Grzeszczuk, Demetri Terzopoulos, and Geoffrey Hinton.  Neuroanimator: Fast neural network emulation and control of physics-based models.  In _Proceedings of the 25th annual conference on Computer graphics and interactive techniques_ , pages 9–20, 1998.
  * Guiasu and Shenitzer (1985) Silviu Guiasu and Abe Shenitzer.  The principle of maximum entropy.  _The mathematical intelligencer_ , 7(1):42–48, 1985.
  * Gurumurthy et al. (2021) Swaminathan Gurumurthy, Shaojie Bai, Zachary Manchester, and J Zico Kolter.  Joint inference and input optimization in equilibrium networks.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Ha et al. (2016) David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Habets (2010) P Habets.  Stabilite asyptotique pour des problemes de perturbations singulieres.  In _Stability Problems_ , pages 2–18. Springer, 2010.
  * Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019.
  * Han et al. (2017) Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu.  Alternating back-propagation for generator network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31, 2017.
  * Harlow (1949) Harry F Harlow.  The formation of learning sets.  _Psychological review_ , 56(1):51, 1949.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In _European conference on computer vision_ , pages 630–645. Springer, 2016.
  * He et al. (2019) Siyu He, Yin Li, Yu Feng, Shirley Ho, Siamak Ravanbakhsh, Wei Chen, and Barnabás Póczos.  Learning to predict the cosmological structure formation.  _Proceedings of the National Academy of Sciences_ , 116(28):13825–13832, 2019.
  * Heess et al. (2015) Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa.  Learning continuous control policies by stochastic value gradients.  In _Advances in Neural Information Processing Systems_ , pages 2944–2952, 2015.
  * Henaff et al. (2019) Mikael Henaff, Alfredo Canziani, and Yann LeCun.  Model-predictive policy learning with uncertainty regularization for driving in dense traffic.  _arXiv preprint arXiv:1901.02705_ , 2019.
  * Higgins et al. (2016) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  _ICLR_ , 2016.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Hochreiter et al. (2001) Sepp Hochreiter, A Steven Younger, and Peter R Conwell.  Learning to learn using gradient descent.  In _International Conference on Artificial Neural Networks_ , pages 87–94. Springer, 2001.
  * Hoffman et al. (2013) Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley.  Stochastic variational inference.  _Journal of Machine Learning Research_ , 14(5), 2013.
  * Hospedales et al. (2020) Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  _arXiv preprint arXiv:2004.05439_ , 2020.
  * Hoyer et al. (2019) Stephan Hoyer, Jascha Sohl-Dickstein, and Sam Greydanus.  Neural reparameterization improves structural optimization.  _arXiv preprint arXiv:1909.04240_ , 2019.
  * Hu et al. (2019) Jiang Hu, Xin Liu, Zaiwen Wen, and Yaxiang Yuan.  A brief introduction to manifold optimization.  _arXiv preprint arXiv:1906.05450_ , 2019.
  * Huang et al. (2022) Tianshu Huang, Tianlong Chen, Sijia Liu, Shiyu Chang, Lisa Amini, and Zhangyang Wang.  Optimizer amalgamation.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=VqzXzA9hjaX.
  * Huszár (2019) Ferenc Huszár.  Notes on imaml: Meta-learning with implicit gradients.  _www.inference.vc_ , 2019.  URL https://www.inference.vc/notes-on-imaml-meta-learning-without-differentiating-through/.
  * Ichnowski et al. (2021) Jeffrey Ichnowski, Paras Jain, Bartolomeo Stellato, Goran Banjac, Michael Luo, Francesco Borrelli, Joseph E Gonzalez, Ion Stoica, and Ken Goldberg.  Accelerating quadratic optimization with reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Jaeger (2002) Herbert Jaeger.  _Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the" echo state network" approach_ , volume 5.  GMD-Forschungszentrum Informationstechnik Bonn, 2002.
  * Jeong and Song (2019) Yeonwoo Jeong and Hyun Oh Song.  Learning discrete and continuous factors of data via alternating disentanglement.  In _International Conference on Machine Learning_ , pages 3091–3099. PMLR, 2019.
  * Jordan et al. (1999) Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul.  An introduction to variational methods for graphical models.  _Machine learning_ , 37(2):183–233, 1999.
  * Karniadakis et al. (2021) George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang.  Physics-informed machine learning.  _Nature Reviews Physics_ , 3(6):422–440, 2021\.
  * Kavukcuoglu et al. (2010) Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun.  Fast inference in sparse coding algorithms with applications to object recognition.  _arXiv preprint arXiv:1010.3467_ , 2010.
  * Kehoe (1988) E James Kehoe.  A layered network model of associative learning: learning to learn and configuration.  _Psychological review_ , 95(4):411, 1988.
  * Khalil et al. (2016) Elias Khalil, Pierre Le Bodic, Le Song, George Nemhauser, and Bistra Dilkina.  Learning to branch in mixed integer programming.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30, 2016.
  * Kim et al. (2018) Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush.  Semi-amortized variational autoencoders.  _arXiv preprint arXiv:1802.02550_ , 2018.
  * Kim (2020) Yoon H Kim.  _Deep latent variable models of natural language_.  PhD thesis, Harvard University, 2020.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma et al. (2019) Diederik P Kingma, Max Welling, et al.  An introduction to variational autoencoders.  _Foundations and Trends® in Machine Learning_ , 12(4):307–392, 2019.
  * Klatte and Kummer (2006) Diethard Klatte and Bernd Kummer.  _Nonsmooth equations in optimization: regularity, calculus, methods and applications_ , volume 60.  Springer Science & Business Media, 2006.
  * Knyazev et al. (2021) Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano.  Parameter prediction for unseen deep architectures.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Kotary et al. (2021) James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder.  End-to-end constrained optimization learning: A survey.  _arXiv preprint arXiv:2103.16378_ , 2021.
  * Kovachki et al. (2021) Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra.  On universal approximation and error bounds for fourier neural operators.  _Journal of Machine Learning Research_ , 22:Art–No, 2021\.
  * Kriváchy et al. (2020) Tamás Kriváchy, Yu Cai, Joseph Bowles, Daniel Cavalcanti, and Nicolas Brunner.  Fast semidefinite programming with feedforward neural networks.  _arXiv preprint arXiv:2011.05785_ , 2020.
  * Ladickỳ et al. (2015) L’ubor Ladickỳ, SoHyeon Jeong, Barbara Solenthaler, Marc Pollefeys, and Markus Gross.  Data-driven fluid simulations using regression forests.  _ACM Transactions on Graphics (TOG)_ , 34(6):1–9, 2015.
  * Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40, 2017.
  * Lambert et al. (2020) Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra.  Objective mismatch in model-based reinforcement learning.  _arXiv preprint arXiv:2002.04523_ , 2020.
  * Lample and Charton (2019) Guillaume Lample and François Charton.  Deep learning for symbolic mathematics.  _arXiv preprint arXiv:1912.01412_ , 2019.
  * Lee et al. (2019a) Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International Conference on Machine Learning_ , pages 3744–3753. PMLR, 2019a.
  * Lee et al. (2019b) Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto.  Meta-learning with differentiable convex optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10657–10665, 2019b.
  * Levine and Abbeel (2014) Sergey Levine and Pieter Abbeel.  Learning neural network policies with guided policy search under unknown dynamics.  In _NIPS_ , volume 27, pages 1071–1079. Citeseer, 2014.
  * Levine and Koltun (2013) Sergey Levine and Vladlen Koltun.  Guided policy search.  In _International Conference on Machine Learning_ , pages 1–9, 2013\.
  * Levine et al. (2016) Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Li and Malik (2016) Ke Li and Jitendra Malik.  Learning to optimize.  _arXiv preprint arXiv:1606.01885_ , 2016.
  * Li and Malik (2017) Ke Li and Jitendra Malik.  Learning to optimize neural nets.  _arXiv preprint arXiv:1703.00441_ , 2017.
  * Li et al. (2020) Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  _arXiv preprint arXiv:2010.08895_ , 2020.
  * Liao et al. (2018) Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard Zemel.  Reviving and improving recurrent back-propagation.  In _International Conference on Machine Learning_ , pages 3082–3091. PMLR, 2018.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Liu et al. (2022) Xinran Liu, Yuzhe Lu, Ali Abbasi, Meiyi Li, Javad Mohammadi, and Soheil Kolouri.  Teaching networks to solve optimization problems, 2022.
  * Lodi and Zarpellon (2017) Andrea Lodi and Giulia Zarpellon.  On learning and branching: a survey.  _Top_ , 25(2):207–236, 2017.
  * Lorraine et al. (2020) Jonathan Lorraine, Paul Vicol, and David Duvenaud.  Optimizing millions of hyperparameters by implicit differentiation.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1540–1552. PMLR, 2020.
  * Lowrey et al. (2018) Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch.  Plan online, learn offline: Efficient learning and exploration via model-based control.  _arXiv preprint arXiv:1811.01848_ , 2018.
  * Luo et al. (2018) Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture optimization.  _arXiv preprint arXiv:1808.07233_ , 2018.
  * Lv et al. (2017) Kaifeng Lv, Shunhua Jiang, and Jian Li.  Learning gradient descent: Better generalization and longer horizons.  In _International Conference on Machine Learning_ , pages 2247–2255. PMLR, 2017.
  * Maclaurin et al. (2015a) Dougal Maclaurin, David Duvenaud, and Ryan Adams.  Gradient-based hyperparameter optimization through reversible learning.  In _International conference on machine learning_ , pages 2113–2122. PMLR, 2015a.
  * Maclaurin et al. (2015b) Dougal Maclaurin, David Duvenaud, and Ryan P Adams.  Autograd: Effortless gradients in numpy.  In _ICML 2015 AutoML workshop_ , volume 238, page 5, 2015b.
  * Marino et al. (2018) Joseph Marino, Yisong Yue, and Stephan Mandt.  Iterative amortized inference.  _arXiv preprint arXiv:1807.09356_ , 2018.
  * Marino et al. (2020) Joseph Marino, Alexandre Piché, Alessandro Davide Ialongo, and Yisong Yue.  Iterative amortized policy optimization.  _arXiv preprint arXiv:2010.10670_ , 2020.
  * Marino (2021) Joseph Louis Marino.  _Learned Feedback & Feedforward Perception & Control_.  PhD thesis, California Institute of Technology, 2021.
  * Marwah et al. (2021) Tanya Marwah, Zachary C Lipton, and Andrej Risteski.  Parametric complexity bounds for approximating pdes with neural networks.  _arXiv preprint arXiv:2103.02138_ , 2021.
  * Meinhardt et al. (2017) Tim Meinhardt, Michael Moller, Caner Hazirbas, and Daniel Cremers.  Learning proximal operators: Using denoising networks for regularizing inverse imaging problems.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1781–1790, 2017.
  * Mena et al. (2018) Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek.  Learning latent permutations with gumbel-sinkhorn networks.  _arXiv preprint arXiv:1802.08665_ , 2018.
  * Merchant et al. (2021) Amil Merchant, Luke Metz, Samuel S Schoenholz, and Ekin D Cubuk.  Learn2hop: Learned optimization on rough landscapes.  In _International Conference on Machine Learning_ , pages 7643–7653. PMLR, 2021.
  * Metz et al. (2016) Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein.  Unrolled generative adversarial networks.  _CoRR_ , abs/1611.02163, 2016.  URL http://arxiv.org/abs/1611.02163.
  * Metz et al. (2019a) Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein.  Understanding and correcting pathologies in the training of learned optimizers.  In _International Conference on Machine Learning_ , pages 4556–4565. PMLR, 2019a.
  * Metz et al. (2019b) Luke Metz, Niru Maheswaranathan, Jonathon Shlens, Jascha Sohl-Dickstein, and Ekin D Cubuk.  Using learned optimizers to make models robust to input noise.  _arXiv preprint arXiv:1906.03367_ , 2019b.
  * Metz et al. (2021) Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman.  Gradients are not all you need.  _arXiv preprint arXiv:2111.05803_ , 2021.
  * Milgrom and Segal (2002) Paul Milgrom and Ilya Segal.  Envelope theorems for arbitrary choice sets.  _Econometrica_ , 70(2):583–601, 2002.
  * Misra et al. (2021) Sidhant Misra, Line Roald, and Yeesian Ng.  Learning for constrained optimization: Identifying optimal active constraint sets.  _INFORMS Journal on Computing_ , 2021.
  * Mnih and Gregor (2014) Andriy Mnih and Karol Gregor.  Neural variational inference and learning in belief networks.  In _International Conference on Machine Learning_ , pages 1791–1799. PMLR, 2014.
  * Monga et al. (2021) Vishal Monga, Yuelong Li, and Yonina C Eldar.  Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing.  _IEEE Signal Processing Magazine_ , 38(2):18–44, 2021.
  * Montgomery and Levine (2016) William H Montgomery and Sergey Levine.  Guided policy search via approximate mirror descent.  _Advances in Neural Information Processing Systems_ , 29:4008–4016, 2016.
  * Nesterov (1983) Yurii Nesterov.  A method for unconstrained convex minimization problem with the rate of convergence o (1/k^ 2).  In _Doklady an ussr_ , volume 269, pages 543–547, 1983.
  * Nichol et al. (2018) Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Olshausen and Field (1996) Bruno A Olshausen and David J Field.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images.  _Nature_ , 381(6583):607–609, 1996.
  * O’donoghue et al. (2016) Brendan O’donoghue, Eric Chu, Neal Parikh, and Stephen Boyd.  Conic optimization via operator splitting and homogeneous self-dual embedding.  _Journal of Optimization Theory and Applications_ , 169(3):1042–1068, 2016.
  * Pan et al. (2020) Xiang Pan, Minghua Chen, Tianyu Zhao, and Steven H Low.  Deepopf: A feasibility-optimized deep neural network approach for ac optimal power flow problems.  _arXiv preprint arXiv:2007.01002_ , 2020.
  * Parikh and Boyd (2014) Neal Parikh and Stephen Boyd.  Proximal algorithms.  _Foundations and Trends in optimization_ , 1(3):127–239, 2014.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32:8026–8037, 2019.
  * Pearlmutter and Siskind (2008) Barak A Pearlmutter and Jeffrey Mark Siskind.  Reverse-mode ad in a functional framework: Lambda the ultimate backpropagator.  _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , 30(2):1–36, 2008.
  * Pennec (2006) Xavier Pennec.  Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements.  _Journal of Mathematical Imaging and Vision_ , 25(1):127–154, 2006.
  * Poli et al. (2020) Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park.  Hypersolvers: Toward fast continuous-depth models.  _arXiv preprint arXiv:2007.09601_ , 2020.
  * Raghu et al. (2019) Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals.  Rapid learning or feature reuse? towards understanding the effectiveness of maml.  _arXiv preprint arXiv:1909.09157_ , 2019.
  * Rajeswaran et al. (2019) Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine.  Meta-learning with implicit gradients.  _Advances in Neural Information Processing Systems_ , 32:113–124, 2019.
  * Ravi and Beatson (2018) Sachin Ravi and Alex Beatson.  Amortized bayesian meta-learning.  In _International Conference on Learning Representations_ , 2018.
  * Ravi and Larochelle (2016) Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  _ICLR_ , 2016.
  * Real et al. (2020) Esteban Real, Chen Liang, David So, and Quoc Le.  Automl-zero: Evolving machine learning algorithms from scratch.  In _International Conference on Machine Learning_ , pages 8007–8019. PMLR, 2020.
  * Rezende and Mohamed (2015) Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In _International conference on machine learning_ , pages 1530–1538. PMLR, 2015.
  * Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  _arXiv preprint arXiv:1401.4082_ , 2014.
  * Richter and Decarlo (1983) Stephen L Richter and Raymond A Decarlo.  Continuation methods: Theory and applications.  _IEEE Transactions on Systems, Man, and Cybernetics_ , SMC-13(4):459–464, 1983.
  * Rick Chang et al. (2017) JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankaranarayanan.  One network to solve them all–solving linear inverse problems using deep projection models.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5888–5897, 2017.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_ , 2017.
  * Runarsson and Jonsson (2000) Thomas Philip Runarsson and Magnus Thor Jonsson.  Evolution and design of distributed learning rules.  In _2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00_ , pages 59–63. IEEE, 2000.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Rusu et al. (2018) Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  _arXiv preprint arXiv:1807.05960_ , 2018.
  * Ryu et al. (2019) Moonkyung Ryu, Yinlam Chow, Ross Anderson, Christian Tjandraatmadja, and Craig Boutilier.  Caql: Continuous action q-learning.  _arXiv preprint arXiv:1909.12397_ , 2019.
  * Salimans and Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=TIdIXIpzhoI.
  * Sanchez-Gonzalez et al. (2020) Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia.  Learning to simulate complex physics with graph networks.  In _International Conference on Machine Learning_ , pages 8459–8468. PMLR, 2020.
  * Santa Cruz et al. (2017) Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould.  Deeppermnet: Visual permutation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3949–3957, 2017.
  * Schmidhuber (1987) Jürgen Schmidhuber.  _Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook_.  PhD thesis, Technische Universität München, 1987.
  * Schmidhuber (1995) Jürgen Schmidhuber.  On learning how to learn learning strategies.  Technical report, TU Munchen, 1995.
  * Sercu et al. (2021) Tom Sercu, Robert Verkuil, Joshua Meier, Brandon Amos, Zeming Lin, Caroline Chen, Jason Liu, Yann LeCun, and Alexander Rives.  Neural potts model.  _bioRxiv_ , 2021.
  * Shaban et al. (2019) Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots.  Truncated back-propagation for bilevel optimization.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 1723–1732. PMLR, 2019.
  * Shao et al. (2021) Zhihui Shao, Jianyi Yang, Cong Shen, and Shaolei Ren.  Learning for robust combinatorial optimization: Algorithm and application.  _arXiv preprint arXiv:2112.10377_ , 2021.
  * Shapiro (2003) Alexander Shapiro.  Sensitivity analysis of generalized equations.  _Journal of Mathematical Sciences_ , 115(4), 2003.
  * Shu (2017) Rui Shu.  Amortized Optimization.  http://ruishu.io/2017/11/07/amortized-optimization/, 2017.  Accessed: 2020-02-02.
  * Silver et al. (2014) David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In _Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32_ , 2014.
  * Sjölund and Bånkestad (2022) Jens Sjölund and Maria Bånkestad.  Graph-based neural acceleration for nonnegative matrix factorization, 2022\.
  * Smola et al. (2007) Alexander J Smola, SVN Vishwanathan, Quoc V Le, et al.  Bundle methods for machine learning.  In _NIPS_ , volume 20, pages 1377–1384, 2007.
  * Stanley et al. (2009) Kenneth O Stanley, David B D’Ambrosio, and Jason Gauci.  A hypercube-based encoding for evolving large-scale neural networks.  _Artificial life_ , 15(2):185–212, 2009.
  * Stellato et al. (2018) Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd.  Osqp: An operator splitting solver for quadratic programs.  In _2018 UKACC 12th international conference on control (CONTROL)_ , pages 339–339. IEEE, 2018.
  * Still (2018) Georg Still.  Lectures on parametric optimization: An introduction.  _Optimization Online_ , 2018.
  * Stuhlmüller et al. (2013) Andreas Stuhlmüller, Jacob Taylor, and Noah Goodman.  Learning stochastic inverses.  _Advances in neural information processing systems_ , 26:3048–3056, 2013.
  * Swersky et al. (2020) Kevin Swersky, Yulia Rubanova, David Dohan, and Kevin Murphy.  Amortized bayesian optimization over discrete spaces.  In _Conference on Uncertainty in Artificial Intelligence_ , pages 769–778. PMLR, 2020.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2818–2826, 2016.
  * Tallec and Ollivier (2017) Corentin Tallec and Yann Ollivier.  Unbiasing truncated backpropagation through time.  _arXiv preprint arXiv:1705.08209_ , 2017.
  * Tennenholtz and Mannor (2019) Guy Tennenholtz and Shie Mannor.  The natural language of actions.  In _International Conference on Machine Learning_ , pages 6196–6205. PMLR, 2019.
  * Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt.  Learning to learn: Introduction and overview.  In _Learning to learn_ , pages 3–17. Springer, 1998.
  * Usman et al. (2021) Ali Usman, Muhammad Rafiq, Muhammad Saeed, Ali Nauman, Andreas Almqvist, and Marcus Liwicki.  Machine learning computational fluid dynamics.  In _2021 Swedish Artificial Intelligence Society Workshop (SAIS)_ , pages 1–4. IEEE, 2021.
  * Van de Wiele et al. (2020) Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih.  Q-learning in enormous action spaces via amortized approximate maximization.  _arXiv preprint arXiv:2001.08116_ , 2020.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Venkatakrishnan et al. (2013) Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg.  Plug-and-play priors for model based reconstruction.  In _2013 IEEE Global Conference on Signal and Information Processing_ , pages 945–948. IEEE, 2013.
  * Venkataraman and Amos (2021) Shobha Venkataraman and Brandon Amos.  Neural fixed-point acceleration for convex optimization.  _arXiv preprint arXiv:2107.10254_ , 2021.
  * Vicol et al. (2021) Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein.  Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies.  In _International Conference on Machine Learning_ , pages 10553–10563. PMLR, 2021.
  * Vilalta and Drissi (2002) Ricardo Vilalta and Youssef Drissi.  A perspective view and survey of meta-learning.  _Artificial intelligence review_ , 18(2):77–95, 2002.
  * Vinuesa and Brunton (2021) Ricardo Vinuesa and Steven L Brunton.  The potential of machine learning to enhance computational fluid dynamics.  _arXiv preprint arXiv:2110.02085_ , 2021.
  * Wainwright and Jordan (2008) Martin J Wainwright and Michael Irwin Jordan.  _Graphical models, exponential families, and variational inference_.  Now Publishers Inc, 2008.
  * Walker and Ni (2011) Homer F Walker and Peng Ni.  Anderson acceleration for fixed-point iterations.  _SIAM Journal on Numerical Analysis_ , 49(4):1715–1735, 2011.
  * Wang et al. (2021) Haoxiang Wang, Han Zhao, and Bo Li.  Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation.  _arXiv preprint arXiv:2106.09017_ , 2021.
  * Wang and Ba (2019) Tingwu Wang and Jimmy Ba.  Exploring model-based planning with policy networks.  _arXiv preprint arXiv:1906.08649_ , 2019.
  * Ward (1937) Lewis B Ward.  Reminiscence and rote learning.  _Psychological Monographs_ , 49(4):i, 1937.
  * Watson and Haftka (1989) Layne T Watson and Raphael T Haftka.  Modern homotopy methods in optimization.  _Computer Methods in Applied Mechanics and Engineering_ , 74(3):289–305, 1989.
  * Webb et al. (2017) Stefan Webb, Adam Golinski, Robert Zinkov, N Siddharth, Tom Rainforth, Yee Whye Teh, and Frank Wood.  Faithful inversion of generative models for effective amortized inference.  _arXiv preprint arXiv:1712.00287_ , 2017.
  * Weng (2018) Lilian Weng.  Meta-learning: Learning to learn fast.  _lilianweng.github.io/lil-log_ , 2018.  URL http://lilianweng.github.io/lil-log/2018/11/29/meta-learning.html.
  * Werbos (1990) Paul J Werbos.  Backpropagation through time: what it does and how to do it.  _Proceedings of the IEEE_ , 78(10):1550–1560, 1990.
  * Wichrowska et al. (2017) Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein.  Learned optimizers that scale and generalize.  In _International Conference on Machine Learning_ , pages 3751–3760. PMLR, 2017.
  * Wiewel et al. (2019) Steffen Wiewel, Moritz Becher, and Nils Thuerey.  Latent space physics: Towards learning the temporal evolution of fluid flow.  In _Computer graphics forum_ , volume 38, pages 71–82. Wiley Online Library, 2019.
  * Wu et al. (2020) Mike Wu, Kristy Choi, Noah Goodman, and Stefano Ermon.  Meta-amortized variational inference and learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 6404–6412, 2020.
  * Wu et al. (2018) Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse.  Understanding short-horizon bias in stochastic meta-optimization.  _arXiv preprint arXiv:1803.02021_ , 2018.
  * Xiao et al. (2021) Yuxin Xiao, Eric P Xing, and Willie Neiswanger.  Amortized auto-tuning: Cost-efficient transfer optimization for hyperparameter recommendation.  _arXiv preprint arXiv:2106.09179_ , 2021.
  * Xie et al. (2020) Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti.  Latent skill planning for exploration and transfer.  In _International Conference on Learning Representations_ , 2020.
  * Xue et al. (2020) Tianju Xue, Alex Beatson, Sigrid Adriaenssens, and Ryan Adams.  Amortized finite element analysis for fast pde-constrained optimization.  In _International Conference on Machine Learning_ , pages 10638–10647. PMLR, 2020.
  * You et al. (2022) Yuning You, Yue Cao, Tianlong Chen, Zhangyang Wang, and Yang Shen.  Bayesian modeling and uncertainty quantification for learning to optimize: What, why, and how.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=EVVadRFRgL7.
  * Zaheer et al. (2017) Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola.  Deep sets.  _Advances in neural information processing systems_ , 30, 2017.
  * Zamzam and Baker (2020) Ahmed S Zamzam and Kyri Baker.  Learning optimal solutions for extremely fast ac optimal power flow.  In _2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_ , pages 1–6. IEEE, 2020.
  * Zeiler (2012) Matthew D Zeiler.  Adadelta: an adaptive learning rate method.  _arXiv preprint arXiv:1212.5701_ , 2012.
  * Zhang and Lesser (2010) Chongjie Zhang and Victor Lesser.  Multi-agent learning with policy prediction.  In _Twenty-fourth AAAI conference on artificial intelligence_ , 2010\.
  * Zhang et al. (2018) Chris Zhang, Mengye Ren, and Raquel Urtasun.  Graph hypernetworks for neural architecture search.  _arXiv preprint arXiv:1810.05749_ , 2018.
  * Zhang et al. (2020) Junzi Zhang, Brendan O’Donoghue, and Stephen Boyd.  Globally convergent type-i anderson acceleration for nonsmooth fixed-point iterations.  _SIAM Journal on Optimization_ , 30(4):3170–3197, 2020.
  * Zhang et al. (2017) Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3929–3938, 2017.
  * Zhang et al. (2019) Xiaojing Zhang, Monimoy Bujarbaruah, and Francesco Borrelli.  Safe and near-optimal policy learning for model predictive control using primal-dual neural networks.  In _2019 American Control Conference (ACC)_ , pages 354–359. IEEE, 2019.
  * Zheng et al. (2022) Wenqing Zheng, Tianlong Chen, Ting-Kuei Hu, and Zhangyang Wang.  Symbolic learning to optimize: Towards interpretability and scalability.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=ef0nInZHKIC.
  * Zhmoginov et al. (2022) Andrey Zhmoginov, Mark Sandler, and Max Vladymyrov.  Hypertransformer: Model generation for supervised and semi-supervised few-shot learning, 2022.