# Recent advances in applying deep reinforcement learning for flow control:
perspectives and future directions

C. Vignon  FLOW, Engineering Mechanics, KTH Royal Institute of Technology,
SE-100 44 Stockholm, Sweden  Mines de Paris - Université PSL, 75005 Paris,
France  J. Rabault  IT Department, Norwegian Meteorological Institute,
Postboks 43, 0313 Oslo, Norway  R. Vinuesa  FLOW, Engineering Mechanics, KTH
Royal Institute of Technology, SE-100 44 Stockholm, Sweden

###### Abstract

Deep reinforcement learning (DRL) has been applied to a variety of problems
during the past decade, and has provided effective control strategies in high-
dimensional and non-linear situations that are challenging to traditional
methods. Flourishing applications now spread out into the field of fluid
dynamics, and specifically of active flow control (AFC). In the community of
AFC, the encouraging results obtained in two-dimensional and chaotic
conditions have raised interest to study increasingly complex flows. In this
review, we first provide a general overview of the reinforcement-learning (RL)
and DRL frameworks, as well as their recent advances. We then focus on the
application of DRL to AFC, highlighting the current limitations of the DRL
algorithms in this field, and suggesting some of the potential upcoming
milestones to reach, as well as open questions that are likely to attract the
attention of the fluid-mechanics community.

##  I Introduction

Deep reinforcement learning (DRL) is a combination of deep learning (DL) and
reinforcement learning (RL). Its ability to solve high-dimensional, non-linear
complex problems has been proven over the past decade in multiple domains such
as, non exhaustively, robotics Kober, Bagnell, and Peters (2013); Gu  _et al._
(2017); OpenAI  _et al._ (2018), language processing Li  _et al._ (2016),
board games (e.g., the game of Go Silver  _et al._ (2017, 2016)), Poker Brown
and Sandholm (2019), video games Mnih  _et al._ (2013, 2015); Szita (2012) and
image analysis Krizhevsky, Sutskever, and Hinton (2012). The acronym DRL is
used when a deep neural network (DNN) takes place in the RL process. DNNs are
well-known for their ability to approximate non-linear functions on high-
dimensional spaces Hornik, Stinchcombe, and White (1989), and have been
introduced in RL methods for that particular purpose. Thanks to these
artificial neural networks (ANNs), the algorithms of the 1990s solving low-
dimensional problems Tesauro (1995); Bagnell and Schneider (2001); Ng  _et
al._ (2004); Peters and Schaal (2006, 2008); Diuk, Cohen, and Littman (2008);
Riedmiller  _et al._ (2009) have been enhanced and now reach super-human
performances in games and can solve a broad range of non-linear, high
dimensional optimization tasks in various domains, as cited above.

Reinforcement-learning methods Sutton and Barto (2018) rely on the interaction
between an agent and an environment. The agent receives observations from the
environment and then decides of an action according to those. Its aim is to
maximize a reward that characterizes the ’goodness’ of the state of the
environment. If the agent interacts with a model of environment instead of the
environment itself, the RL algorithm is said ’model-based’, otherwise if the
agent interacts directly with the environment, the RL algorithm is ’model-
free’. In that case, the algorithm only uses the partial observations of the
environment as inputs, and returns the actions as an output. Such algorithms
are thus considered as ’black-box’ methods, because an analytical description
of the system is not needed to control it.

Problems including chaotic (turbulent) flows are usually non-linear and high-
dimensional, due to the governing Navier–Stokes equations and the continuous
environment space, i.e. the flow. Efficient ’black-box’ RL methods can
therefore be useful and fruitful in that field, as traditional analysis
methods based on local linearization usually struggle in this context. In Ref.
Brunton and Noack, 2015, the authors review the main methods of closed-loop RL
adapted to the specific case of fluid dynamics and their inherent challenges,
i.e. high-dimensionality of the environment and/or action space(s), and
complex non-linearity of the governing equations. The application of RL
methods and derivatives in fluid mechanics has known a flourishing growth for
a few years. One may refer to recent advances with late reviews on DRL Garnier
_et al._ (2021); Vinuesa  _et al._ (2022), and more general machine-learning
(ML) methods applied to fluid dynamics Pino  _et al._ (2022). The use of RL in
fluid mechanics has various purposes, such as (non-exhaustively): subgridscale
Novati, de Laroussilhe, and Koumoutsakos (2021); Kurz, Offenhäuser, and Beck
(2023), wall Bae and Koumoutsakos (2022) and turbulence Beck, Flad, and Munz
(2019, 2019); Beck and Kurz (2021) modelling, RL-augmented computational fluid
dynamics (CFD) solvers Kurz  _et al._ (2022a, b), the shape optimization of
airfoils Lampton, Niksch, and Valasek (2008, 2010); Ghraieb  _et al._ (2021),
the active control of the separation in a flow surrounding an aircraft Batikh,
Baldas, and Colin (2017), the reduction of the drag past cars Noack  _et al._
(2021) and bluff bodies in both laminar Rabault  _et al._ (2019) and chaotic
regimes Tang  _et al._ (2020); Ren, Rabault, and Tang (2021); Varela  _et al._
(2022), the study of the wake behind a cylinder Li and Zhang (2021) or the
reduction of the skin-friction drag in a turbulent channel Sonoda  _et al._
(2022); Guastoni  _et al._ (2023). Within the non-RL machine-learning methods,
genetic programming Koza (1992) (GP) has shown comparative results in multiple
applications Gautier  _et al._ (2015); Li  _et al._ (2017); Fan, Zhou, and
Noack (2019); Ren, Wang, and Tang (2019); Li  _et al._ (2020). For the sake of
brevity, as an entire review could be dedicated to those, GP algorithms will
be briefly introduced in the present review solely to compare their main
properties with the DRL ones.

In fluid mechanics, two fields are therefore mainly studied for the
application of RL: shape optimization and flow control. Within the former, a
body is immersed in a flow, and the RL algorithm modifies its shape in order
to optimize some characteristic due to the dynamic of the fluid (e.g., the
drag and/or the lift – see for instance Ref. Ghraieb  _et al._ , 2022). A
classical example is the reduction of the drag coefficient behind a body in a
laminar or turbulent flow: in Refs. Lampton, Niksch, and Valasek, 2008, 2010
this idea is applied to the shape optimization of airfoils (more recent
studies on that subject are given in Refs. Ghraieb  _et al._ , 2021; Viquerat
_et al._ , 2020, using DRL). Flow control aims at actively influencing the
behavior of a flow, such as controlling separation, turbulence or heat
transfers, in order to reach a more desirable state. It is a subject of great
interest from both societal and economical points of view Ghraieb  _et al._
(2021). Flow control strategies can be passive or active Gad-el Hak (1996).
Contrarily to the passive solutions (e.g. riblets and vortex generators Scott
Collis  _et al._ (2004)), the active methods need a supply in energy to
perform the control sequence. As they could achieve unrivalled performances,
DRL algorithms applied to active flow control problems are largely studied for
their promising engineering prospects (see, e.g., Refs. Brunton and Noack,
2015; Vinuesa  _et al._ , 2022).

This review is aimed at presenting the current challenges in the field of
active flow control (AFC), particularly when the control is based on DRL
methods. To this end, a general overview of the main reinforcement-learning
processes is first given in section II, and then follows a presentation of the
recent advances in deep reinforcement learning (section III). Eventually,
applications of machine learning – and specifically DRL – in AFC are developed
in section IV. Attention will be focused on the challenges inherent to the
field of fluid dynamics, and some possible upcoming milestones for the
coupling of DRL and AFC will be suggested accordingly.

##  II Reinforcement Learning

###  II.1 Introduction

Reinforcement learning (RL) is a branch of data-driven methods, considered as
a subsection of machine learning, which is itself a subset of artificial
intelligence. The first study developing the modern form of reinforcement
learning was provided by Sutton (1988) at the end of the 1980s. In his time,
Sutton (1988) merged two distinct approaches: the learning by trials and
errors, notably studied for its natural applications in the learning process
of animals, and optimal control. By combining the two, the aim of modern RL is
to optimize the decisions taken by an agent in a surrounding environment
during a sequence of actions. In other words, the idea is to train and improve
a control law (the agent) able to optimize a target system (the environment)
by taking sequences of actions.

Reinforcement learning has already been applied to many fields with successful
results. During the past decade, RL algorithms have been enhanced thanks to
neural networks, developing DRL methods on the basis of RL. One may refer to
Ref. Kober, Bagnell, and Peters, 2013 for RL and Ref. OpenAI  _et al._ , 2018
for DRL applied to robotics, to Refs. Mnih  _et al._ , 2013, 2015 for DRL
applied to Atari games with super-human performances, to Ref. Szita, 2012 for
video games and Ref. Silver  _et al._ , 2016 for Go game, to name a few.

In reinforcement learning, an agent interacts with an environment in a three-
steps sequence :

  1. 1.

The environment gives to the agent a partial observation $o$ of its state $s$.
In a continuous environment space, the observation is necessarily incomplete.
Thus, the observation $o$ may not provide enough information to describe $s$.
In order to overcome this issue, the state $s_{t}$ obtained at time $t$ is
often described by the succession of actions and observations that led to it :
$s_{t}~{}=~{}(s_{0},a_{1},o_{1},a_{2},...,o_{t-1},a_{t})$, or similarly:
$s_{t}~{}=~{}(s_{0},a_{1},s_{1},a_{2},...,s_{t-1},a_{t})$.

  2. 2.

According to the observation $o$ of state $s$, the agent takes an action $a$,
following a certain policy $\pi$: $a=\pi(s)$. This action impacts the
environment and modifies its state. As a consequence, its new state is now
denoted $s^{\prime}$ (a priori different from $s$).

  3. 3.

Eventually, a reward $r$ is given to the agent, characterizing the ’goodness’
of the action taken according to the old state $s$ and the new one
$s^{\prime}$.

Figure 1: Interactions between an agent and an environment in a generic
reinforcement-learning loop.

Through these three interactions, which take place at each time-step of a
global process, the RL algorithm gathers information and progressively
enhances the reward. To this end, the reward defines the feature that the user
wants to improve. Instead of the instantaneous reward, introduced as $r$ above
(or more precisely $r_{t}$ at time $t$), RL algorithms usually optimize a
cumulative reward, denoted by $R$ and defined as:

$R=\sum\limits_{{t=0}}^{T}\gamma^{t}r_{t}\text{ },$ (1)

where $\gamma$ is the discount factor (usually, $\gamma\gtrsim 0.99$), and $T$
is the duration of an episode. The sequence of the algorithm is as follows:

  * •

First, at $t=0$, the state of the environment is (randomly) initialized (state
$s_{0}$).

  * •

At each time-step $t$, the three steps process described above (see Fig.1) is
applied, returning a reward $r_{t}$ after having changed the state of the
environment from $s_{t}$ to $s_{t+1}$, by enforcing the action $a_{t}$
according to the policy $\pi$.

  * •

When $t$ reaches $T$, the process stops. The sequence of actions running
during a time $T$ is called an episode. At the end of an episode, the
cumulative reward $R$ is computed and the policy modified with the long-term
objective of optimizing $R$. During an episode, $\lfloor\frac{T}{t}\rfloor$
actions are taken. In the cumulative reward, actions with a low instantaneous
reward can be compensated by others, no distinction is made apart from the
discounted factor $\gamma^{t}$. This factor slightly favours short-term rather
than long-term strategies, while enabling $R$ to converge when episodes with
numerous actions are considered. A full sequence
$(s_{0},a_{1},s_{1},a_{2},...,s_{T})$ is called a trajectory, and further
represented by $\tau$.

The cumulative reward is commonly not computed over the whole trajectory, but
from a certain time-step $t_{0}$. Equation (1) is then replaced by:

$R=\sum\limits_{{t=t_{0}}}^{T}\gamma^{t-t_{0}}r_{t}~{}.$ (2)

Depending on the representation chosen for the environment, RL algorithms are
ordinarily separated into two broad categories. If the agent interacts with an
artificial modelling of environment, the algorithm is considered to be model-
based, otherwise the algorithm is model-free. In section II.2, examples of
model-based methods are presented, but since the majority of DRL algorithms
are model-free, attention will mainly be given to the model-free category in
this review. Additionally, the reader can refer to the works of Jaderberg  _et
al._ (2017) and Ha and Schmidhuber (2018), or to the recent review of Rabault
and Kuhnle (2022), for examples of model-free algorithms incorporating a
model-learning process. By trying to build a representation of the environment
during the learning process and benefit from it, these initially-model-free
methods are at the cross-section of model-free and model-based algorithms.

In the following subsections we will only discuss the main concepts and
features of RL algorithms, without a focus on DRL. How NNs can be used
together with RL algorithms to develop DRL will be discussed in section III.

###  II.2 Model-based methods

Model-based methods require an artificial representation of the environment.
Historically, linear models first spread across the field of flow control, as
they were relatively simple and proved to be efficient on multiple problems,
e.g the stabilization of convectively unstable flows Barbagallo  _et al._
(2012); Dahan, Morgans, and Lardeau (2012); Gautier and Aider (2014). But
flows are also commonly characterized by their strongly nonlinear behaviors.
Although linear models are justifiable in turbulent situations where the flow
behaves like a linear amplifier Brackston  _et al._ (2016), their application
to highly turbulent regimes is ambiguous and has long been debated in the
community (see, e.g., Ref. Kim and Bewley, 2007 and the references therein).
To that purpose, Brackston  _et al._ (2016) developed a (nonlinear) stochastic
model, inspired from the work of Rigas  _et al._ (2015) and governed by the
nonlinear Langevin equation. A feedback controller coupled to this model
successfully stabilized the wake behind a bluff body immersed in a three
dimensional turbulent flow. The drag was efficiently reduced by suppressing
the large-scale structures. In their review, Rowley and Dawson (2017) analyze
the robustness of reduced order models when applied to linear and nonlinear
problems. These model-based methods aim at extracting the main dynamic
processes that structure the flow, in order to simplify its modelling. The
reader may also refer to the works of Queipo  _et al._ (2005) and Koziel  _et
al._ (2016) for examples of model-based applications in aerodynamics.

The computational gain obtained with model-based methods results from the
simplification of the environment. This gain is especially relevant in the
situation of flow control, but necessarily leads to sub-optimal solutions when
it comes to high-Reynolds-numbers regimes, as the models only partially
approximate the turbulences. Hence, models can still be configured for these
ranges of regime, but one is then confronted to a mediation between sub-
optimal complex model-based controllers that rely on necessarily approximate
models of the flow, and computationally-heavy ’black-box’ model-free
controllers that directly interact with a Navier-Stokes solver. Recent model-
based methods, such as the Probabilistic Ensembles with Trajectory Sampling
(PETS) Chua  _et al._ (2018), try to compete with the efficient modern model-
free deep RL algorithms and thus go along the direction of developing both
model-based and model-free methods.

###  II.3 Model-free methods

Model-free methods are increasingly popular in the community because of how
relatively simple they are to apply. Indeed, the agent directly interacts with
the environment: no assumption has to be made on the environment modelling.
Thanks to regular partial observations of the environment state, the agent
finds a control sequence $(a_{0},...,a_{t},...,a_{T})$ that maximizes the
cumulative reward.

The environment is usually considered as stochastic (i.e. when the action $a$
is applied on the state $s$, the new state cannot be predicted and deduced
exactly) and is commonly modelled by a Markov decision process (MDP) Bellman
(1957); Howard (1960); Sutton and Barto (2018), even though it could be
represented by other stochastic processes Rabault  _et al._ (2019), or
considered to be deterministic Silver  _et al._ (2014).

####  II.3.1 Markov Decision Process

Markov decision processes (MDPs) Bellman (1957); Howard (1960) are at the core
of modern reinforcement learning Sutton and Barto (2018). A MDP is a tuple
$(S,A,\mathcal{P}_{a},R)$ with $S$ and $A$ the sets of (respectively) all the
possible states and actions, and $R$ the set of all the possible rewards. One
can define $\mathcal{P}_{a}(s,s^{\prime})$ as the probability that taking
action $a$ in state $s$ will lead to state $s^{\prime}$. Reusing the standard
nomenclature proposed in Ref. Garnier  _et al._ , 2021, let us define the
operator $\mathbb{P}$ as:

$\displaystyle\mathbb{P}:S\times A\times S\longrightarrow[0;1]\text{ };$
$\displaystyle(s,a,s^{\prime})\mapsto\mathbb{P}(s^{\prime}|s,a)=\mathcal{P}_{a}(s,s^{\prime})~{}.$
(3)

Hence, $\mathbb{P}$ represents the probability of obtaining state $s^{\prime}$
after having taken action $a$ in state $s$. The policy is defined by:

$\displaystyle\pi_{\theta}:S\longrightarrow A\text{ };$ $\displaystyle
s\mapsto\pi_{\theta}(s)~{}.$ (4)

Note that $\pi_{\theta}(s)$ returns the actions probability distribution when
the initial state is $s$. The action is then chosen by the agent according to
$\pi_{\theta}(s)$, and $\theta$ represents the set of parameters that
characterizes $\pi$. The aim is to find the policy $\pi$ that maximizes the
cumulative reward $R$. Modifications of the policy are made by choosing
different sets of parameters $\theta$. Furthermore, $\mathbb{P}$ and
$\pi_{\theta}$ completely describe the fundamental process of the RL
algorithm: given a state $s$, the agent takes the action $a$ according to
$\pi_{\theta}(s)$, and then $\mathbb{P}$ returns the probability of obtaining
a new state $s^{\prime}$.

####  II.3.2 State value and state-action value functions

In addition, three functions are usually defined in order to describe and
classify the RL algorithms. The state-action value function, also called the
$Q$-function, is defined by:

$Q^{\pi}(s,a)=\mathbb{E}_{\tau\sim\pi}\left[R(\tau)|s_{0}=s,a_{0}=a\right]\text{
},$ (5)

and the state value function is defined by:

$V^{\pi}(s)=\mathbb{E}_{\tau\sim\pi}\left[R(\tau)|s_{0}=s\right]\text{ }.$ (6)

Hence, $Q^{\pi}(s,a)$ corresponds to the expected cumulative reward when the
action $a$ is taken in the initial state $s$ and then follows the trajectory
$\tau$ according to the policy $\pi$, whereas $V^{\pi}(s)$ corresponds to the
expected cumulative reward when the initial state is $s$ and then follows the
trajectory $\tau$ according to the policy $\pi$. Thus:

$V^{\pi}(s)=\mathbb{E}_{\tau\sim\pi}[Q^{\pi}(s,\pi(s))]~{}.$ (7)

Eventually, the advantage function $A^{\pi}$ is defined by:

$A^{\pi}=Q^{\pi}-V^{\pi}~{}.$ (8)

$A^{\pi}$ characterizes the advantage (in terms of cumulative reward) of
taking action $a$ in state $s$ rather than taking any other possible action.

Among the model-free RL algorithms, two main methods are both developed for
their promising performances: value-based Watkins (1989) and policy-based
methods Williams (1992). During the RL process, the agent takes actions
according to the state of the environment by following a policy. With policy-
based methods, the aim is to directly optimize the policy $\pi$ (see section
II.3.6). Differently, value-based methods rely on learning the Q-function, and
then deriving the optimal policy from it (see section II.3.5).

####  II.3.3 Closed-loop and open-loop methods

Additionally to the value/policy criterion, RL methods are also commonly
distinguished according to the frequency of the interactions between the agent
and the environment. In closed-loop control, the agent obtains a regular
feedback of the current state of the environment thanks to sensors. The RL
algorithm uses these measurements to adapt its policy during the episode, in
order to tend to a more desired state (see Ref. Brunton and Noack, 2015 for a
review on closed-loop turbulence control). On the other hand, open-loop
methods assume that the decision policy does not rely on the state of the
environment, or at least that a steady or periodic actuation can modify the
principal dynamical processes of the environment Meliga, Sipp, and Chomaz
(2010). One may refer to the recent works of Shahrabi (2019) and Ghraieb  _et
al._ (2021) for the use of open-loop methods in flow control. By not
considering the surrounding environment, these methods are generally less
efficient than the closed-loop solutions, but easier to implement. The
upcoming sections will mainly focus on closed-loop algorithms, as they are the
most promising and challenging methods for the future.

####  II.3.4 On-policy and off-policy online and offline methods

Another distinction among RL algorithms is made between on-policy and off-
policy methods. Within the former, the agent solely learns about the policy it
enforces to the environment, contrarily to the latter where the agent can
learn from other policies. For instance, Degris, White, and Sutton (2012)
execute a second (behavior) policy aimed at choosing the trajectories while
the first one decides the actions. Another example of off-policy method is the
Q-learning process Watkins and Dayan (1992) mentioned in the next section.
Furthermore, with on-policy methods, the data collected before the last update
of the (single) policy cannot be used for forthcoming updates, as these data
had been generated by a policy that differs from the last version. Therefore,
off-policy settings are of great interest, as they enable the algorithm to
reuse previous experiences that may be stored from either earlier in the
learning process or completely separate learning runs, and to learn multiple
tasks in parallel thanks to the different policies.

Offline RL algorithms learn and update their policy(-ies) only at the end of
an episode, or after N episodes, while online algorithms make updates ’in real
time’. Online strategies present the advantage to learn from the current state
of the environment instead of past situations, but require demanding
computational properties Sutton and Whitehead (1993), not always available nor
achievable. The reader may refer to the work of Degris, Pilarski, and Sutton
(2012) and the references therein for additional information on the
differences between offline and online methods.

####  II.3.5 Value-based methods

Value-based methods rely on the Q-function. The optimal Q-function can be
defined as $Q^{*}~{}=~{}\text{max}_{\pi}Q^{\pi}$. In reinforcement learning,
the objective of value-based methods is to have a Q-table filled with the
values of $Q^{*}(s,a)$ for any $(s,a)$ in $S\times A$. In other words, the aim
is to be able to know the best policy to apply when starting from any tuple
$(s,a)$ in $S\times A$, in order to maximize the expected cumulative reward.
The Q-table can be thought of as an array or a transition reward matrix in
discrete states case, or can be approximated by a NN or another function
approximator in general. For simplicity, in this section Q is considered as an
array that can be filled with values, while an ANN trained by gradient descent
will be considered in deep Q-learning (see section III.2.1).

It is not possible to directly compute $Q^{*}$, contrarily to $Q^{\pi}$. Thus,
the Q-table is first filled with values of $Q^{\pi}$, estimated by exploring
the space $S\times A$. Additionally, $Q^{*}$ respects the Bellman optimally
condition Bellman and Dreyfus (1962):

$\displaystyle Q^{*}(s,a)=r(s,a)+\sum\limits_{{s^{\prime}\in
S}}\mathbb{P}(s^{\prime}|s,a)\gamma\text{max}_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})$
$\displaystyle=r(s,a)+\mathbb{E}_{s^{\prime}\in
S}\left[\gamma\text{max}_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})\right]~{}.$
(9)

Here, $r(s,a)$ represents the instantaneous reward returned when action $a$ is
taken in state $s$. This relationship on $Q^{*}$ means that when the action
$a$ is first applied to the initial state $s$, leading to the state
$s^{\prime}$ (as it is a stochastic process, $s^{\prime}$ is not known
precisely, hence the expectation on $s^{\prime}$), if the optimal Q-function
is known at the next time-step, then the best policy is to choose the next
action $a^{\prime}$ that maximizes $Q^{*}(s^{\prime},a^{\prime})$.

To update the Q-table, Eq. (II.3.5) inspires the recursive update relationship
given below:

$Q^{\pi}(s,a)\longleftarrow\mathbb{E}_{s^{\prime}\in
S}\left[r(s,a)+\gamma\text{max}_{a^{\prime}\in
A}Q^{\pi}(s^{\prime},a^{\prime})\right]$ (10)

It has been proved that updates of $Q^{\pi}$ by following (10) make $Q^{\pi}$
converge to $Q^{*}$ Bellman and Dreyfus (1962).

Eventually, when the Q-table is filled with the estimated values of $Q^{*}$
(thanks to the recursive process given by Eq. (10)), and given a state $s$,
one can then optimize the action that the agent will take, by choosing the
action $a^{*}$ according to the ’greedy’ policy:

$a^{*}=\text{argmax}_{a\in A}Q^{*}(a,s)~{}.$ (11)

These value-based methods are usually called Q-learning Watkins and Dayan
(1992). One can refer to the original work of Watkins (1989), which was
already an evolution of difference learning Sutton (1988).

####  II.3.6 Policy-based methods

The second main category of RL methods directly relies on an estimation and an
optimization of the policy function $\pi_{\theta}$, and does not consider an
optimization of the $Q$-function. These policy-based methods have been
developed since – at least – the work of Williams (1992). The objective
function $J$ is introduced in order to evaluate the quality of the policy. As
the long-term aim still remains to maximize the expected cumulative reward,
$J$ is commonly defined by:

$J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R(\tau)\right]~{}.$ (12)

$J$ can vary thanks to modifications of $\theta$, the set of parameters
characterizing the policy. Thus, the objective is to find:

$\theta^{*}=\text{argmax}_{\theta}J(\theta)~{}.$ (13)

To that purpose, the gradient of $J$ is introduced, so that $\theta$ can be
recursively modified by following the direction of this gradient:

$\theta\longrightarrow\theta+\lambda\nabla_{\theta}J(\theta)~{},$ (14)

with $\lambda$ being a positive parameter. Two distinct options are discussed
for the evaluation of $\nabla_{\theta}J$, whether the policy is considered to
be stochastic or deterministic, i.e. whether $\pi(s)$ returns a probability
distribution of actions $a$ or an exact and unique value $a$. In the former
case, the gradient is computed according to the relationship Garnier  _et al._
(2021); Sutton  _et al._ (1999); Williams (1992):

$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum\limits_{{t=t_{0}}}^{T}\nabla_{\theta}\text{log}(\pi_{\theta}(a_{t}|s_{t}))R(\tau)\right]~{}.$
(15)

In this stochastic approach, the ascent of $J$ consists in first sampling the
policy on different trajectories, then computing an average of the cumulative
rewards obtained on these trajectories (the average thus replaces the
expectation in Eq. (15)), and finally adjusting $\theta$ with Eq. (14), in
order to progressively enhance the cumulative reward.

In the deterministic approach (deterministic policy-gradient algorithm, or DPG
algorithm Silver  _et al._ (2014)), the environment is still represented with
a stochastic process (MDP), but the policy is considered to be deterministic.
To keep in mind the difference, the deterministic policy is now represented by
$\mu_{\theta}$. Replacing $\pi_{\theta}$ with $\mu_{\theta}$, the objective of
the deterministic policy gradient algorithms is to apply the same principle as
in the stochastic case. The mathematical formulation of the deterministic
policy gradient is beyond the purpose of the present work. One may refer to
the original work of Silver  _et al._ (2014), and bear in mind that with a
deterministic approach it is not needed to sample over a range of actions and
states, but only over a range of states, as the actions are deterministically
deduced.

###  II.4 Limitations of the classical reinforcement-learning methods

Classical RL methods, i.e. algorithms that use simple function approximators
such as matrices, polynomials or other simple functions, may encounter
limitations. Historically, applications of value-based methods to real-world
problems Tesauro and Sejnowski (1989); Crites and Barto (1995) have rapidly
been overtaken by faster policy-based methods (see, e.g. the policy gradient
method Sutton  _et al._ (2000)), which quickly proved their efficiency and
applicability to various real-world tasks Bagnell and Schneider (2001); Ng
_et al._ (2004); Peters and Schaal (2006).

Through the idea of experience replay (see the original work of Lin (1992)),
value-based RL methods have known a revival (e.g., Ref. Riedmiller,
Montemerlo, and Dahlkamp, 2007 for an application). The experience replay
technique largely shrinks the number of interactions between the agent and the
environment and thus accelerates these methods. Nevertheless, both the value-
based and policy-based RL methods remain limited to problems with low state
and action spaces Mnih  _et al._ (2015); Degris, White, and Sutton (2012).

##  III Deep reinforcement learning

Deep reinforcement learning (DRL) methods consist in the combination of neural
networks and classical RL algorithms. Neural networks are known to be
universal function approximators Hornik, Stinchcombe, and White (1989), able
to treat high-dimensional and non-linear problems, and thus to overcome the
limitations set by the simple function approximators used in traditional RL.
This means that, given a sufficient amount of example data representing a well
behaved (typically, continuous) function, and a large enough ANN, the ANN can
represent the function with arbitrary precision, i.e. ANNs large enough are
dense in the space of continuous functions. Their application in reinforcement
learning led to breakthroughs in various fields as they largely enhanced the
performances of the previous methods in multi-input multi-output (MIMO)
problems. To cite a few, the DRL algorithm developed by Mnih  _et al._ (2015)
obtained super-human levels in Atari games, and the one created by Silver  _et
al._ (2016) won against the European champion of Go five times in a row (to
contextualize this achievement, no algorithm was able to beat a professional
human player in this particularly complex board game before). Eventually, the
PPO algorithm implemented by Rabault  _et al._ (2019) successfully led to the
first control of a flow by DRL.

In this section, a short introduction to artificial neural networks is first
provided. An overview of the main DRL methods – presented in light of their RL
precursors – is then suggested. Within these methods, the most recent and
encouraging developments are presented, as well as the challenges they face
and try to overcome.

###  III.1 Artificial neural networks

Artificial neural networks (ANNs, or NNs) are function approximators, based on
connections between singular units, called neurons. A neuron is an entity
characterized by three features : a set of weights $w=(w_{1},...,w_{n})$, with
$n\in\mathbb{N}^{*}$ corresponding to the number of inputs connected to the
neuron, a bias $b$ ($b\in\mathbb{R}$), and the activation function $\sigma$,
an hyper-parameter chosen at the conception of the neural network. Taking a
vector $x=(x_{1},...,x_{n})$ as an input, the neuron returns the scalar
$\sigma(w.x+b)$. Neurons are usually organized in successive layers ; we then
talk about deep NNs (DNNs). Each neuron of a layer is connected to one or more
neurons of the precedent and next layers. When each neuron of each layer is
connected to all the neurons of the precedent and of the next layer, the NN is
said fully connected. FCANNs (fully connected artificial neural networks) are
commonly used in DRL. A schematic of FCANN is given in figure 2. This
illustrative example is constituted of a layer of two output neurons (red
disks), and two layers of five neurons (green disks). The first layer on the
left of the figure is not constituted of neurons: the purple disks correspond
to the inputs (here of size three). They are conventionally represented
similarly to neurons but simply represent a scalar input (or, equivalently, a
neuron with $n=1$, $b=0$, $w_{1}=1$ and $\sigma=\text{id}_{\mathbb{R}}$). The
black lines correspond to the connections between the neurons.

Figure 2: Schematic of a fully-connected artificial neural network. Purple
denotes input, red output and green the hidden layers.

The training of an ANN consists in tuning all its degrees of freedom, i.e
tuning the biases and weights of all the neurons so that the ANN accurately
represents a goal function known through examples. To do so, these parameters
are commonly tuned step by step, by following, e.g., the back-propagating
algorithm LeCun, Bengio, and Hinton (2015).

ANNs can be used in RL algorithms for their ability to estimate complex non-
linear functions Hornik, Stinchcombe, and White (1989). The number of NNs and
the functions they approximate within the RL algorithm define several
different DRL methods, which we analyze and compare in the following
paragraphs.

###  III.2 Value-based methods

####  III.2.1 Deep Q-learning

Artificial neural networks are commonly known for their ability to deal with
high-dimensionality and non-linearity, following their universal approximator
properties, as previously discussed. Therefore, instead of filling a Q-table
with the values of $Q^{\pi}$ – as it was done in value-based RL –, ANNs may be
appropriate to directly approximate $Q^{*}$.

The approximation of $Q^{*}$ by the neural network is represented by
$Q_{\theta}$, with all the weights and biases of the neural network
represented by the set $\theta$. The objective of the training phase of the
neural network is to tune $\theta$ in order to achieve the best approximation
of $Q^{*}$. To do so, a loss function is defined as the mean-squared error of
the Bellman equation (Eq. (II.3.5)):

$\displaystyle
L(\theta)=\mathbb{E}_{s,a}\bigg{[}\frac{1}{2}\Big{\\{}\mathbb{E}_{s^{\prime}}\left[r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})\right]-Q_{\theta}(s,a)\Big{\\}}^{2}\bigg{]}$
$\displaystyle=\mathbb{E}_{s,a,s^{\prime}}\left[\frac{1}{2}\left\\{r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})-Q_{\theta}(s,a)\right\\}^{2}\right]$
$\displaystyle+\mathbb{E}_{s,a,s^{\prime}}\left[\frac{1}{2}\mathbb{V}_{s^{\prime}}\left[r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})\right]\right].$
(16)

The variance of
$r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})$ being
independent of the weights $\theta$, the second term is usually ignored Mnih
_et al._ (2015), and the loss function limited to:

$L(\theta)=\mathbb{E}_{s,a,s^{\prime}}\bigg{[}\frac{1}{2}\Big{\\{}r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})-Q_{\theta}(s,a)\Big{\\}}^{2}\bigg{]}~{}.$
(17)

More precisely, at each time-step i:

$L_{i}(\theta_{i})=\mathbb{E}_{s,a,s^{\prime}}\bigg{[}\frac{1}{2}\Big{\\{}r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta_{i-1}}(s^{\prime},a^{\prime})-Q_{\theta_{i}}(s,a)\Big{\\}}^{2}\bigg{]}~{}.$
(18)

The objective of the neural network is to minimize the loss function, thus to
modify $Q_{\theta_{i}}$ to get it closer to the target
$r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta_{i-1}}(s^{\prime},a^{\prime})$,
computed with the weights $\theta_{i-1}$ of the precedent update. To this end,
the update:

$\theta_{i+1}\longleftarrow\theta_{i}-\alpha\nabla_{\theta_{i}}L_{i}(\theta_{i})~{},$
(19)

is applied (with $\alpha$ a positive parameter) and should make $Q_{\theta}$
converge to $Q^{*}$.

However, a naive coupling – such as the one mentioned above – between an ANN
and a RL algorithm would be compromised by two issues that may lead to
divergence. First, the process of learning from sequences of actions and
observations on-policy induces correlations. Indeed, the algorithm is
potentially biased if it solely learns from the successions of tuples
$(s_{t},a_{t},s_{t+1},r_{t})$ that occur during the running episode (on-
policy), because these tuples are generally not independent within an episode,
which has a tendency to make the NN training fail. Furthermore, a minor
modification of the value of $Q$ may change significantly the data
distribution and thus may lead to an unstable algorithm.

In order to tune these two biases, Mnih  _et al._ (2013) suggest the
integration of a replay buffer in their algorithm. Filled with a certain
amount of tuples $(s_{t},a_{t},s_{t+1},r_{t})$ that occurred in past episodes,
the replay buffer enables the ANN to have more uncorrelated input data and,
simultaneously, a more stabilized data distribution, which is required for a
correct learning process. By adapting the experience replay technique
suggested in Ref. Lin, 1992, Mnih  _et al._ (2013) thereby propose the Deep
Q-Learning (DQL) algorithm, which masters complex control policies in Atari
games. Schaul  _et al._ (2015) extend the idea of replay buffer by suggesting
to optimize its filling, by favouring less frequent actions instead of a
uniform process, as the learning phase could benefit from these unusual
events.

Even with the DQL algorithm, a major restriction remains, due to the fact that
the neural network is chasing a moving target. As developed above, the ANN
tries to reach the Bellman optimally condition (Eq. (II.3.5)) by minimizing
the loss function $L$. To do so, it modifies $Q_{\theta_{i}}$ to get it closer
to the target
$r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta_{i-1}}(s^{\prime},a^{\prime})$,
but the target itself changes at each timestep $i$. This issue induces
instability and divergence in many situations, notably when the estimation of
Q is provided by a nonlinear function approximator Tsitsiklis and Van Roy
(1997), such as a FCANN. This instability thus highly reduces the algorithm
robustness.

####  III.2.2 Deep Q-networks

Developed by Mnih  _et al._ (2015), Deep Q-networks (DQNs) are an evolution of
DQL. The DQN method mainly improves the ability of the neural network to treat
high-dimensional environment spaces and suppresses the issue of divergence
detailed above, by providing an innovation to the DQL algorithm.

The problem of divergence is solved by exploiting a second ANN, a ’slow-
reacting clone’ of the original one. Let us define the loss function with Eq.
(17). In DQL, the algorithm tries to equalize the target
$r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})$ with
$Q_{\theta}(s,a)$, by using a single neural network. In DQN, an ANN is applied
for the computation of $Q_{\theta}$ and the ’slow-reacting’ copy is applied
for the estimation of the target. Denoting $\theta^{\prime}$ as the set of
weights and biases of this second network, the loss function is then defined
by:

$L(\theta)=\mathbb{E}_{s,a,s^{\prime}}\bigg{[}\frac{1}{2}\Big{\\{}r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime})-Q_{\theta}(s,a)\Big{\\}}^{2}\bigg{]}~{}.$
(20)

The strategy suggested by Mnih  _et al._ (2015) is to regularly equalize
$\theta$ and $\theta^{\prime}$. If such an equalization is made at each update
of $\theta$, then the method becomes identical to DQL. Different strategies
have been developed to correlate $\theta$ and $\theta^{\prime}$. In Ref.
Lillicrap  _et al._ , 2015, the authors develop a different method (called
DDPG and detailed below), but use the same idea, and propose the iterative
process $\theta^{\prime}\longleftarrow\tau\theta+\left(1-\tau\right)\theta$,
with $\tau\ll 1$. In the original DQN algorithm Mnih  _et al._ (2015),
$\theta^{\prime}$ is actualized one time over N updates of $\theta$
($N\in\mathbb{N}^{*}\gg 1$). These strategies with two ANNs thus eliminate the
instability by slowing down the movements of the target.

Dueling DQN Wang  _et al._ (2015) is an evolution of DQN in which the neural
network is split into two streams, one aimed at evaluating the advantage
function, while the other one computes the state value function $V^{\pi}$. The
Q-function is then estimated by combining the approximations of $A^{\pi}$ and
$V^{\pi}$ (in a non-trivial manner). The advantage function depends on the
actions chosen whereas the state value function does not. By taking advantage
of these two functions and their different dependencies on the actions, the
dueling DQN algorithm overcomes the DQN method particularly in situations
where there are many similar-valued actions.

####  III.2.3 Double deep Q-learning and double deep Q-networks

Double DQL and double DQN Van Hasselt (2010); van Hasselt, Guez, and Silver
(2015) have been developed in order to solve the issue of overestimation
happening in DQL and DQN, a problem partially unknown or ignored up to then.
The overestimation in these algorithms is due to the fact that in the
computation of the target, the same neural network is used both for the
selection of the action and for its evaluation. Indeed, in DQL, one can define
the target $Y$ as:

$\displaystyle
Y(s,a)=r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime})$
$\displaystyle=r(s,a)+\gamma
Q_{\theta}(s^{\prime},\text{argmax}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime}))~{},$
(21)

and likewise in DQN:

$\displaystyle
Y(s,a)=r(s,a)+\gamma\text{max}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime})$
$\displaystyle=r(s,a)+\gamma
Q_{\theta^{\prime}}(s^{\prime},\text{argmax}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime}))~{}.$
(22)

In both cases, the neural network that estimates $Q^{*}$ is used to choose
$a^{\prime}$ and then to apply $Q^{*}(s^{\prime},a^{\prime})$, which can lead
to an overestimation of the expected cumulative reward. Not only is
overestimation quite common, but it can also adversely affect the performances
of the algorithm van Hasselt, Guez, and Silver (2015). For the DQL algorithm,
a new method with a second network is presented in Ref. van Hasselt, Guez, and
Silver, 2015, inspired by previous works Van Hasselt (2010). In double DQL,
two networks are exploited so that one focuses on the choice of the action,
and the other one on the evaluation. Hence, the target becomes:

$Y(s,a)=r(s,a)+\gamma
Q_{\theta^{bis}}(s^{\prime},\text{argmax}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime}))~{},$
(23)

with $\theta^{bis}$ the set of weights and biases of the new neural network,
used for the evaluation. The two networks regularly invert their roles, so
that $\theta$ and $\theta^{bis}$ are similarly updated. The double DQN
algorithm does not require any other network, a second one being already
exploited within the DQN method. Indeed, van Hasselt, Guez, and Silver (2015)
suggest:

$Y(s,a)=r(s,a)+\gamma
Q_{\theta^{\prime}}(s^{\prime},\text{argmax}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime}))~{}.$
(24)

According to van Hasselt, Guez, and Silver (2015), the performances of the
value-based DRL methods are enhanced, even the already robust performances of
the DQN algorithm in Atari games Mnih  _et al._ (2015).

####  III.2.4 Challenges and openings

The DQN algorithm has first been developed to stabilize the DQL method, and
notably enhanced its efficiency to deal with large neural networks. Thanks to
that, Deep Q-Networks have shown great results with high-dimensional
observation spaces Mnih  _et al._ (2015). But DQN or even double DQN
algorithms are not easily able to deal with high-dimensional action spaces.
Indeed, as they seek $\text{argmax}_{a}Q_{\theta}(.,a)$ at each timestep,
increasing the dimensionality of the actions space highly reduces their
performances. Some methods have been developed to overcome this issue, making
continuous action spaces possible. Based on a combination of policy gradient
and Q-function, they also offer larger possibilities, as they may consider
either off-policy or on-policy strategies, whereas the methods mentioned above
are limited to off-policy strategies Mnih  _et al._ (2016), due to the replay
buffer. These promising policy-based methods are presented in the next
section.

###  III.3 Policy-gradient and actor-critic methods

####  III.3.1 Deep policy gradient

Amidst the current most promising DRL methods, the second main contender is
the family of actor-critic methods. These are obtained by combining elements
of Q-learning and policy-gradient methods, which we will discuss now.

The deep policy gradient algorithm is a policy-based method, i.e it directly
optimizes the policy (see section II.3.6). Explicitly, an ANN approximates the
policy $\pi_{\theta}$, with $\theta$ representing its set of weights and
biases. The loss function is defined by:

$L(\theta)=-\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum\limits_{{t=t_{0}}}^{T}\text{log}(\pi_{\theta}(a_{t}|s_{t}))R(\tau)\right]~{},$
(25)

so that $\nabla_{\theta}L(\theta)=-\nabla_{\theta}J(\theta)$ (see Eq. (15) for
$\nabla_{\theta}J$). This expected value is approximated by an average over a
set of trajectories. $R(\tau)$ is commonly replaced with the advantage
function $A^{\pi_{\theta}}$, as it reduces the variance of the expectation
Schulman  _et al._ (2015) and thus reduces the number of trajectories
necessary to well average this expectation. Hence, in the deep policy gradient
algorithm, the loss function:

$L(\theta)=-\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum\limits_{{t=t_{0}}}^{T}\text{log}(\pi_{\theta}(a_{t}|s_{t}))A^{\pi_{\theta}}(s_{t},a_{t})\right]~{},$
(26)

is usually considered. Nonetheless, as the expectation is averaged over a set
of trajectories at the end of an episode, the algorithm does not differentiate
between the most accurate actions and the worse, but only computes the reward
obtained from the whole trajectory. Actor-critic methods adapt the deep policy
gradient algorithm to deal with this issue.

####  III.3.2 Advantage actor-critic methods

Deep policy gradient methods are limited as they do not differentiate positive
from negative actions. To clear it up, actor-critic methods rely on a
combination of policy-based and value-based methods, as one neural network
(the actor) approximates $\pi_{\theta}$, the policy by which the actions are
chosen, and a second one (the critic) computes the Q-function
$Q_{\theta^{\prime}}$, in order to evaluate the goodness of the action taken
by the agent. With the same prospect of variance reduction,
$Q_{\theta^{\prime}}$ is often replaced with $A_{\theta^{\prime}}$, an
estimation of the advantage function. The method is then called Advantage
Actor-Critic (A2C) Sutton and Barto (2018).

One may refer to the review of Garnier  _et al._ (2021) for explicit
implementations of the vanilla deep policy gradient and A2C algorithms. Many
variations have been developed, such as the asynchronous A2C (A3C) suggested
in Ref. Mnih  _et al._ , 2016. The latter was presented to have similar - if
not better - performances compared to the DQN algorithm in Atari games, and
additionally to be able to treat continuous action spaces, which was a reason
for the development of policy-based methods. Indeed, as an $argmax_{a\in A}()$
function is not computed anymore, these methods enable one to tremendously
increase the actions space $A$, up to a continuous space in some algorithms.

####  III.3.3 The deep deterministic policy gradient method

As explained above, a major limit of DQN and other value-based methods is
their lack of robustness with high-dimensional action spaces. The deep
deterministic policy gradient (DDPG) algorithm was first developed by
Lillicrap  _et al._ (2015) to solve this specific issue. It is a model-free
off-policy actor-critic method, inspired by the Deterministic Policy Gradient
(DPG) algorithm presented in Ref. Silver  _et al._ , 2014 (see section
II.3.6). Contrarily to the previously presented DRL methods, the DDPG (like
the DPG) algorithm considers a deterministic policy mapping, and not
stochastic. Thus the actions space is explored thanks to a perturbative
method:

$a_{t}=\mu_{\theta+N_{1}}\left(s_{t}\right)+N_{2}~{},$ (27)

with $N_{1}$ and $N_{2}$ two different noisy processes, $\theta$ the set of
weights and biases of the actor, and $a_{t}$ the action taken at timestep $t$
in state $s_{t}$ according to the deterministic policy $\mu$. The critic
computes the optimal Q-function similarly to the DQN algorithm, but replaces
the $argmax_{a\in A}()$-seeking with a gradient over $\mu$ (see Ref. Lillicrap
_et al._ , 2015), and thus enables one to leverage high-dimensional and
continuous action spaces.

Bucci  _et al._ (2019) succeeded in stabilizing the dynamics of a chaotic
system governed by the one-dimensional Kuramoto-Sivashinsky equation thanks to
a DDPG controller, while more recently Sonoda  _et al._ (2022) applied this
algorithm to a flow control problem (the reduction of the skin friction drag
in a channel flow). Both of these examples illustrate the promising
performances of the DDPG method with high-dimensional action spaces.

####  III.3.4 Proximal policy optimization

The proximal policy optimization (PPO) algorithm is an on-policy actor-critic
method, developed by Schulman  _et al._ (2017) in order to deal with the lack
of robustness of the DQN Mnih  _et al._ (2015) and ’vanilla’ policy gradient
Mnih  _et al._ (2016) methods, and simplify the efficient-but-complex trust
region policy optimization (TRPO) algorithm Schulman  _et al._ (2015), not
discussed in the present review.

Apart from the particular DDPG algorithm, the policy gradient and actor-critic
methods usually aim at optimizing the loss function defined in Eq. (26), now
denoted by $L^{PG}(\theta)$. To do so, these algorithms proceed over several
epochs of gradient ascent, i.e they alternate a first phase where they fill a
batch of samples (a sample being a sequence $(s_{t},a_{t},s_{t+1},r_{t})$ for
instance), and a second phase in which the expectation of the gradient of
$L^{PG}(\theta)$ is averaged over the batch, and then the neural network
parameters $\theta$ updated following the direction of this estimation. The
idea behind the TRPO and PPO algorithms is identical, but $L^{PG}$ is replaced
with other functions. For instance, the surrogate loss:

$\displaystyle
L^{\textsc{Clip}}(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\Bigg{[}\sum\limits_{{t=t_{0}}}^{T}\text{min}\big{\\{}r_{t}(\theta)A^{\pi_{\theta}}(s_{t},a_{t});$
$\displaystyle\text{clip}(r_{t}(\tau),1-\epsilon,1+\epsilon)A^{\pi_{\theta}}(s_{t},a_{t})\big{\\}}\Bigg{]}~{},$
(28)

is suggested in Ref. Schulman  _et al._ , 2017 for the PPO algorithm, with:

$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta^{\text{old}}}(a_{t}|s_{t})}~{},$
(29)

being a factor that penalizes an update of $\theta$ that would induce an
important modification of the policy (comparatively to the situation before
the update, with the previous set of NN parameters $\theta^{\text{old}}$).
$\epsilon$ is a parameter chosen between 0 and 1, and
$\text{clip}(r_{t}(\tau),~{}1~{}-\epsilon,1+\epsilon)$ returns $r_{t}(\tau)$
if its value is in $[1-\epsilon;1+\epsilon]$, $1+\epsilon$ if it is greater
than this limit, and $1-\epsilon$ otherwise. In the TRPO algorithm, the
objective function is similarly penalized according to the size of the update
Schulman  _et al._ (2015). The PPO algorithm is a simplification of this
previous work as it only considers a first order optimization process. Another
variant of loss function is proposed in Ref. Schulman  _et al._ , 2017, using
a KL-penalty Kakade and Langford (2002).

After having tuned $\epsilon=0.2$, Schulman  _et al._ (2017) observe that the
PPO algorithm outperforms most of the previous online policy gradient
algorithms (A2C and vanilla PG algorithms Mnih  _et al._ (2016), a combination
of A2C and trust region algorithm Wang  _et al._ (2016), the TRPO method
Schulman  _et al._ (2015) and the cross-entropy method (CEM) Szita and Lörincz
(2006)), on almost all the continuous control environments.

Hence, the PPO algorithm is one of the most promising modern DRL methods, and
has already successfully been applied to a variety of problems (see, e.g.,
Ref. Peng  _et al._ , 2018). It is often considered as the ’state-of-the-art’
for continuous control Ren, Wang, and Tang (2021). To cite an application in
flow control, Rabault  _et al._ (2019) used a PPO algorithm for performing the
first active flow control combining CFD simulations and deep reinforcement
learning, in the standard example of the cylinder immersed into a 2D flow, and
Ren, Rabault, and Tang (2021) extended the previous work to a weakly turbulent
regime.

Variations on the basis of the original PPO algorithm now flourish, such as
PPO-1 Ghraieb  _et al._ (2021). PPO-1 is a degenerate Viquerat  _et al._
(2020) version of the PPO method, developed for open-loop control. The
algorithm runs multiple simulations in parallel, shuffles the data coming from
all the environments and fills several mini-batches with these mixed data,
then uses these mini-batches sequentially to update the NN, and eventually
repeats the sequence by launching multiple parallel simulations again, etc.
Another variant is the PPO with Covariance Matrix Adaptation (PPO-CMA)
Hämäläinen  _et al._ (2018), an algorithm able to dynamically expand the
variance of the exploration policy to speed up the seeking phase, and then to
shrink it when the algorithm gets close to the global optimum. PPO is stable
but sometimes prematurely reduces the exploration variance, and then may
converge to local optima, hence the development of this algorithm. The reader
may also refer to the Maximum a posteriori Policy Optimization (MPO) algorithm
Abdolmaleki  _et al._ (2018a) and extensions Abdolmaleki  _et al._ (2018b) for
similar motivations.

A recent variation of PPO-CMA, called AS-PPO-CMA Paris, Beneddine, and Dandois
(2023), aims at reducing the number of actuators while limiting the loss
induced in the cumulative reward. This development goes along with the
prospect of increasing the complexity of the problems handled by DRL, where an
effective reduction of the actions space may be critical.

For comparison, a non-comprehensive overview of RL methods is suggested in
figure 3, while schematics representing the main different DRL architectures
are gathered in figure 4.

Figure 3: Non-exhaustive classification of machine learning methods, with
focus on the reinforcement-learning-based techniques. Figure 4: Comparison
between (a) RL, (b) DRL, (c) policy-based DRL, (d) value-based DRL, (e)
advantage actor-critic DRL and (f) DQN (simplified) architectures.

###  III.4 Parallelization

Multiple processes have been developed to reduce the computational cost of DRL
algorithms, in order to then consider more complex tasks. The ones gathered by
the idea of parallelization are presented in this section.

First, when it comes to fluid dynamics problems that combine CFD simulations
with DRL algorithms, one may consider the parallelization of the simulations
themselves. This subject has a long history Simon (1992); Gropp and Smith
(1990), and well-implemented parallel CFD solvers nowadays reach limitations
correlated to the number of nodes Gropp  _et al._ (2001), hence limiting the
gain of time with parallel CFD simulations.

Consequently, as the computational cost due to the simulations may remain
important even with parallel CFD solvers, the parallelization of the DRL
algorithms is of great interest. Speedups would enable one to study more
challenging flow configurations, which are the main objective of the current
researches in the field. Rabault and Kuhnle (2019) consider the simple bi-
dimensional benchmark presented in Ref. Rabault  _et al._ , 2019 and adapt the
PPO algorithm to parallelization. Their idea is to run independent episodes on
several parallel environments, so that the collection of data used for the
training phase is parallelized. Indeed, they prove that if one runs $N$
episodes sequentially between two updates of the neural network, running these
$N$ episodes in parallel is equivalent, and thus divides the computational
cost by approximately $N$. Results are conclusive, the computational time
being divided by twenty in their most optimal implementation. This is of great
interest, as problems with higher complexity usually need to increase the size
of the batch and the number of episodes between updates of the NN. Increasing
the complexity would therefore no longer induce an increase of the
computational cost, if additional parallel environments were added
accordingly. The idea of PPO-1 Ghraieb  _et al._ (2021) is to achieve multiple
simulations in parallel, then to fill several mini-batches with the shuffled
data, and eventually to sequentially use the mini-batches to update the NN.
The parallelization process is thus very similar to the one of Rabault and
Kuhnle (2019).

Massively distributed architectures on multiple GPUs have been implemented to
accelerate DRL methods, such as DQN with the Gorila framework Nair  _et al._
(2015). Multiple agents act in parallel on identical copies of the
environment, have their own replay buffer and compute the loss function
similarly to the DQN process (Eq. (20)). A central actor merges the
information from all the agents, and then enforces the same ANN update to each
of them. With 100 parallel agents, Gorila outperformed DQN on many Atari
games, with computations approximately twenty times faster. A similar process
has also been suggested by Ong, Chavez, and Hong (2015), whereas Mnih  _et
al._ (2016) implemented the same principle but with multiple CPU threads on a
single machine, thereby outperforming the previous methods while offering the
possibility to have stable value-based and policy-based, on-policy and off-
policy efficient strategies. Indeed, an important improvement brought by the
parallel agents is that it stabilizes value-based methods even without any
replay buffer Mnih  _et al._ (2016), thus allowing more efficient on-policy
methods.

Eventually, Belus  _et al._ (2019) exploit for the first time the
translational invariance and the locality of the control problem in a DRL
algorithm, in order to enhance its efficiency. Their work is not a
parallelization process but also enables one to reach more complex problems,
and it will be more extensively discussed in the last section.

##  IV Deep reinforcement learning for flow control

###  IV.1 Introduction passive and active control

Flow control strategies are usually separated in two categories Gad-el Hak
(1996): passive and active methods. The former include, e.g., riblets and
vortex generators Scott Collis  _et al._ (2004) and other LEBUs (large-eddy-
break-up devices) Alfredsson and Örlü (2018). They do not need a supply in
energy nor a feedback from the environment. Their robustness, low
manufacturing costs and global simplicity have notably enabled them to be
applied in real-world systems, contrarily to the active methods which are
often complex to implement concretely. But the active methods present the
advantage to be generally more efficient, and in a wider range of operating
conditions. For instance, while riblets used for drag reduction in turbulent
boundary layers can perform a 5 to 9% drag reduction Bechert and Bartenwerfer
(1989), modern active methods can perform reductions twice as much important,
or even more (see Ref. Scott Collis  _et al._ , 2004, or Ref. Vinuesa  _et
al._ , 2022 for a recent review on flow control in turbulent boundary layers).

Passive methods are intrinsically open-loop as they do not consider the
evolution of the surrounding environment. However, active methods can be
either open-loop or closed-loop. Open-loop active methods, by not updating the
actuators with a potential feedback of the environment state, are sub-optimal.
But having a good understanding of the open-loop behavior is almost a
prerequisite to the further development of closed-loop actuators Scott Collis
_et al._ (2004). For instance, actuators used to control flow separation have
first proved to be efficient in periodic regimes Greenblatt and Wygnanski
(2000); Seifert, Darabi, and Wyganski (1996) and paved the way to modern
closed-loop actuators (see Ref. Rabault  _et al._ , 2019 for an example of
closed-loop flow control using synthetic jets).

For historical and comprehensive reviews on active and passive methods, the
reader may refer to the works of Gad-el Hak, Pollard, and Bonnet (1998), Moin
and Bewley (1994) and Lumley and Blossey (1998), and to the review of Pino
_et al._ (2022) for the use of machine learning methods in flow control. The
objective of this section is to give a general overview of the last advances
in active flow control (AFC), and more particularly in the application of the
DRL methods to that field. The current challenges and upcoming milestones in
the combination of DRL and AFC are highlighted.

###  IV.2 Active flow control

Active flow control has grown at the frontier of various traditional fields,
including fluid mechanics, control theories and machine learning Bewley
(2001). Thus, it has inherently been confronted to terminology conflicts
Sutton, Barto, and Williams (1992); Bersini and Gorrini (1996); Bewley (2001);
Recht (2019); Nian, Liu, and Huang (2020), but has developed fruitful hybrid
methods during the past two decades. This interdisciplinary effort has indeed
been carried out with the prospect of many promising applications Pino  _et
al._ (2022); Brunton and Noack (2015), such as delaying the transition past
airfoils/aircraft wings Kühn  _et al._ (2013a); Batikh, Baldas, and Colin
(2017), reducing the drag coefficient past bluff bodies (see, e.g. Ref. Ren,
Wang, and Tang, 2021) or controlling convective heat transport Beintema  _et
al._ (2020), to cite a few.

Contrarily to the passive methods, the active control of a flow needs a supply
in energy. In addition, the data used for the control can be provided by
experimental results or CFD simulations. These data can be used during the
control sequence (reactive control) or only before (predetermined control). As
feed-forward methods are not considered in the present review, active reactive
feedback methods will simply be called feedback control thereafter (see
Fig.5). In terms of terminology reconciliation, open-loop and closed-loop
(D)RL methods applied to the control of a flow are respectively assimilated to
predetermined and feedback control. The closed-loop DRL methods are thus an
active reactive feedback data-driven strategy (see Fig.5).

Figure 5: Non-exhaustive classification of flow-control strategies.

####  IV.2.1 Predetermined control

The predetermined control of a flow is regardless of its instantaneous state.
The control sequence is determined in advance, and then applied without any
feedback. It facilitates the control, since no sensors are needed and no
feedback loop has to be implemented Sonoda  _et al._ (2022).

In a numerical approach, Jung, Mangiavacchi, and Akhavan (1992) and then
Quadrio and Ricco (2004) have enforced spanwise oscillations at the walls of a
fully developed turbulent channel, and computed reductions of the turbulent
drag of the order of 40%. Several studies Atzori  _et al._ (2020, 2021);
Fahland  _et al._ (2021) have also considered predetermined control based on
blowing and suction in turbulent wings, also leading to over $10\%$
improvement in aerodynamic efficiency. Alongside these direct numerical
simulation approaches, experimental versions of the spanwise oscillating walls
have been implemented Laadhari, Skandaji, and Morel (1994); Choi (2002);
Karniadakis and Choi (2003), and important but lower values (25-35%) of drag
reduction were observed. Quadrio and Ricco (2004) also reported a net energy
saving of 7% (taking into account the power necessary for the wall
oscillations), hence highlighting that even without a feedback algorithm,
sensors or complex actuators, a predetermined control method can be efficient.

Other predetermined methods have been tested in the field of flow control. One
may refer to Refs. Min  _et al._ , 2006; Lieu, Moarref, and Jovanović, 2010
for similar but streamwise blowing and suction, to Refs. Kametani and
Fukagata, 2011; Kühn  _et al._ , 2013a for uniform blowing and suction, and to
Ref. Kühn  _et al._ , 2013b for a periodic blowing on a wing flap (high-lift
device). All these studies illustrate the efficiency of actuators used in a
steady or cyclic manner with open-loop algorithms. Nevertheless, as they do
not take into account the evolution of the dynamics in the flow, their range
of efficiency remains limited. Quadrio and Ricco (2004) explain that the input
parameters of the oscillations depend on the Reynolds number for instance. In
complex turbulent flows, predetermined methods thus encounter limitations
similar to the passive methods.

####  IV.2.2 Feedback control

Feedback control includes a closed-loop process to take into account the
instantaneous evolution of the flow. It is sometimes called reactive – or even
briefly active – control. The number of actuators, their degrees of freedom
and the quantity of input data – collected by one or more sensors – can make
these methods challenging. Within these methods, multiple processes can be
distinguished. Opposition control methods Choi, Moin, and Kim (1994), optimal
control theory (see, e.g., Ref. Sargent, 2000), genetic programming Koza
(1992), reinforcement learning Sutton (1988) and deep reinforcement learning
Sutton and Barto (2018) applied to AFC problems are successively discussed in
the present section, and compared in view of the milestones to reach. If well
implemented, the feedback should enhance the efficiency of the control
sequence, by adapting itself to the ’live’ state of the flow. For instance,
Ren, Wang, and Tang (2019) obtained an encouraging 20% improvement of the
control performances by applying a closed-loop process rather than any open-
loop method in the AFC problem they were considering.

Opposition control

has first been developed by Choi, Moin, and Kim (1994) to reduce the drag
generated by the skin friction in a fully developed turbulent channel flow, by
trying to annihilate the coherent structures in the boundary layer with a
feedback process. These coherent structures play a preponderant role in the
transport of turbulences Cantwell (1981); Robinson (1991), and such a drag
reduction would have numerous applications, since this type of wall-bounded
flows are classical in engineering problems. The idea behind opposition
control is to apply local blowing and suction at the wall according to a
sensing detection plane, parallel to the wall and typically located at
$h^{+}=15$ wall units from the wall Hammond, Bewley, and Moin (1998). For
instance, if a normal speed of $v(x,y,h^{+})$ is detected at the position
$(x,y,h^{+})$ in the detection plane, a normal blowing (or suction, according
to the sign of $v$) of $-\alpha v(x,y,h^{+})$ is applied at the point
$(x,y,0)$ of the wall (with $\alpha>0$ the linear coefficient). By controlling
either the normal (as presented above) and/or spanwise velocities in such a
simple feedback control, a skin friction reduction of approximately 20-25% is
obtained Choi, Moin, and Kim (1994); Chung and Tariq (2011). These methods are
thus simple and show relatively efficient results, but they become unusable in
more complex flows. Indeed, in the standard situation presented above, the
relationship between the detected speed and the actuation is considered to be
linear, and the optimal coefficient ($\alpha$) is found by trial and error.
But this trial and error process is quite inefficient, even in the linear
approximation Sonoda  _et al._ (2022). Additionally, the drag reduction rate
obtained by opposition control in turbulent boundary layers decreases with
increasing Reynolds numbers Stroh  _et al._ (2015). Since then, the opposition
control method is not aligned with the current objectives of controlling more
sophisticated and turbulent flow conditions.

Optimal control theory

can efficiently optimize an input parameter, even with large degrees of
freedom, by directly tackling the mathematical equations of the flow, and
showed successful results in AFC in a weakly turbulent channel configuration
Bewley, Moin, and Temam (2001). In optimal control, the dynamical evolution of
the system is explicitly developed until a certain time horizon, and then
’rewound’ back to the initial state in order to solve the adjoint equations
and compute a certain cost function. The optimal value of the parameter at
stake is then tuned according to the cost function. As the non-negligible time
horizon and the resolution of the adjoint equations require important
computation time and resources, a sub-optimal theory as been developed Lee,
Kim, and Choi (1998); Hasegawa and Kasagi (2011), carrying the strategy of
setting an arbitrarily small time horizon (thus suppressing the necessity to
solve the adjoint equations Sonoda  _et al._ (2022)) while limiting the
deterioration of the optimization. But sub-optimal control solutions are
limited. Indeed the more complex and turbulent the flow becomes, the more
longer the time horizon needs to be, in order to have a good understanding of
the dynamics of the system (for a further optimization of the parameter at
stake). Additionally, the time horizon cannot be extended endlessly in optimal
control, the adjoint equations being unstable beyond a certain threshold Wang,
Hu, and Blonigan (2014). Consequently, in prospect of (actively) controlling
flows with high Reynolds numbers, the optimal control theory may present non-
negligible shortcomings.

Data-driven methods

are promising, since they could overcome the previous flaws encountered within
the opposition and optimal control methods. They take various shapes but are
gathered by the same preeminence of/need for consequent data.

• The Bayesian optimization (BO) and Lipschitz global optimization (LIPO)
processes are ’black-box’ global optimization techniques Pino  _et al._
(2022). Compared to the upcoming genetic programming and RL methods, they
present the advantage to be less complex, as they are (respectively) linear
and quadratic function approximators, and to require less computational time.
In the classical BO method, the function to optimize (e.g. the cumulative
reward) is modeled by a Gaussian process. Given a certain data set, one can
thus obtain a probability distribution of the values taken by the function at
stake. In the BO iterative process, a function suggesting where to sample next
is also defined. The LIPO method relies on the same basis, but with a
different surrogate function instead of the Gaussian process. One can refer to
the work of Pino  _et al._ (2022) for a more extensive and complete
formulation of the BO and LIPO methods applied to AFC, and to the work of
Blanchard  _et al._ (2021) for the first application of open-loop BO in AFC.

• A genetic programming control algorithm is a nature-inspired feedback
process that relies on the succession of populations Koza (1992). A population
of individuals (i.e. a set of control laws) is first generated, applied to the
environment (i.e. a numerical simulation or experiment per control law is
run), and then multiple processes (replication, cross-over, mutation amidst
the ’individuals’, i.e. the control laws) modify the previous population to
produce a new one, and gradually optimize a function (e.g. the cumulative
reward). Examples of successful GP applications in AFC can be found in Ref.
Gautier  _et al._ , 2015 for a separation control issue, in Ref. Li  _et al._
, 2017 for a drag-reduction-behind-a-car problem, in Ref. Fan, Zhou, and
Noack, 2019 for the mixing optimization of a turbulent jet, in Ref. Li  _et
al._ , 2020 for an example of open-loop GP control, or more quantitatively in
the review proposed by Noack (2019). The aforementioned applications used
experimental setups to feed the closed-loop process with feedback data. Ren,
Wang, and Tang (2019) present an innovative combination of CFD and GP-based
AFC to suppress the vorticity induced by a vibrating cylinder. Their results
outperform the overall open-loop methods by approximately 20%, and thus arouse
encouraging prospects.

• Eventually, the application of DRL in AFC has already shown promising
results, and overtook the majority of the results obtained with previous
methods. With the objective of increasing the complexity of the handled flow
configurations, DRL methods appear to be the most relevant contender. As a
consequence, a specific section is dedicated to these methods (see section
IV.3).

By applying PPO to laminar flow conditions ($Re=100$), Tokarev, Palkin, and
Mullyadzhanov (2020) obtain performances comparable to the ones achieved with
an adjoint-based method in Ref. Flinois and Colonius, 2015. However, in the
situation of a wall-bounded channel flow, and even at a low friction Reynolds
number (of about 180), the DRL algorithm developed by Guastoni  _et al._
(2023) outperforms the opposition control method by over 20 percentage points,
with drag reductions of, respectively, 46% and 20%. Furthermore, opposition
and optimal control reach limitations when it comes to complex tasks such as
controlling high-Reynolds turbulent flows, because the design of a relevant
explicit control law becomes challenging, if not impossible. Pino  _et al._
(2022) additionally illustrate that GP and DDPG outperform BO and LIPO on
three showcases with progressive difficulty. Nonetheless, as the generality of
the function approximator increases, the number of episodes required for the
training phase grows in proportion. Therefore, it may worth combining GP and
DRL algorithms with ’simpler’ methods Pino  _et al._ (2022); Li  _et al._
(2019); Mendez  _et al._ (2023), or associating model-free DRL algorithms with
environments partially modeled thanks to the physical understanding of the
dynamics of the system (see, e.g., Refs. Li and Zhang, 2021; Qin  _et al._ ,
2021 and the references therein). Compared to DRL, GP presents the advantage
of performing better with fewer sensors Castellanos  _et al._ (2021). However,
DRL methods are inherently more suitable for multi-input multi-output problems
than GP Ren, Wang, and Tang (2021). The last section of this review will
therefore be devoted to the application of DRL to AFC.

###  IV.3 Deep reinforcement learning and active flow control: challenges and
prospects

Within the field of fluid dynamics, the deep reinforcement learning methods
can be applied to a variety of problems. Historically, the first one
considered consisted in the behavioral study of swimmers in shoal formations
(see Refs. Gazzola  _et al._ , 2016; Novati  _et al._ , 2017; Verma, Novati,
and Koumoutsakos, 2018 and related works), where the DRL algorithm could
choose the position and movements of one/multiple swimmer(s) placed in the
wake of a leader. Not only did those studies enhance the scientific knowledge
on complex energy saving processes, such as the interception and optimization
of vortices by the followers, but it may also have engineering repercussions
and benefits in real-world applications Whittlesey, Liska, and Dabiri (2010).
Hence, these successful results gave a first overview of the capability and
efficiency of DRL algorithms to handle complex physical problems, and paved
the way to multiple applications in the field of fluid dynamics Rabault  _et
al._ (2020); Vinuesa and Brunton (2022); Brunton, Noack, and Koumoutsakos
(2020); Li and Zhang (2021).

Nowadays, numerous DRL agents have been implemented in order to, e.g.,
optimize the position of a small static (e.g., Ref. Garnier  _et al._ , 2021)
or moving (e.g., Refs. Fan  _et al._ , 2020; Xu  _et al._ , 2020) body around
a larger one with the prospect of reducing the recirculation bubble/the drag
downstream, to directly modify the shape of the principal body Jiao  _et al._
(2021); Viquerat  _et al._ (2020); Yan  _et al._ (2019); Yonekura and Hattori
(2019) or to control its movements Tokarev, Palkin, and Mullyadzhanov (2020);
Ren, Wang, and Tang (2021) with the same objective of optimizing its
aerodynamic/hydrodynamic properties. Other research works handle heat
transport issues such as the Rayleigh-Bénard instability Beintema  _et al._
(2020), or other convectively-unstable flows Xu and Zhang (2023).

Rabault  _et al._ (2019) led the first study applying a DRL algorithm (PPO) to
an AFC problem. Two small synthetic jets were placed symmetrically on the top
and bottom of a cylinder immersed in a 2D flow, aimed at shrinking the
instabilities behind the body (i.e. the Kármán vortex street). This bi-
dimensional showcase, inspired by the original setup of Schäfer  _et al._
(1996), is now a generic benchmark used to test the recent developments of DRL
in the field, and has therefore known great and recurrent interest Rabault and
Kuhnle (2019); Tang  _et al._ (2020); Ren, Rabault, and Tang (2021); Varela
_et al._ (2022). Other studies also consider AFC problems using actuators, to
control the flow separation Shimomura  _et al._ (2020); Batikh, Baldas, and
Colin (2017) or to limit the skin friction in wall-bounded channels Sonoda
_et al._ (2022); Guastoni  _et al._ (2023), to cite a few.

The community now redoubles its efforts on increasing the complexity of the
AFC problems handled with DRL. The first upcoming milestone consists in
proving the efficiency of the existing algorithms when it comes to highly
turbulent conditions. Issues of instabilities in turbulent boundary layers Xu,
Lombard, and Sherwin (2017); Xu  _et al._ (2017) (TBLs) would for instance
become reachable – then with fruitful engineering prospects (see, e.g., Ref.
Yu  _et al._ , 2021 for the control of a TBL with genetic algorithms). For
this purpose, successive research works have for instance increased the
Reynolds number in the originally-laminar cylinder problem treated in Ref.
Rabault  _et al._ , 2019 with a Reynolds number of $Re=100$. They successfully
proved the efficiency of PPO in a range $Re\in[60;400]$ Tang  _et al._ (2020),
then for the first time in a weakly turbulent condition ($Re=1000$) Ren,
Rabault, and Tang (2021) and recently reached $Re=2000$ Varela  _et al._
(2022). In Ref. Varela  _et al._ , 2022, the algorithm suggests a strategy
that differs significantly from the opposition-control solution. PPO is often
chosen for its relative simplicity while having comparable performances to the
other DRL algorithms, and because flow control issues generally require
continuous actuation (thus not reachable with DQN for instance). But DDPG also
raised promising results. In Ref. Bucci  _et al._ , 2019, the authors
stabilized a chaotic system governed by the 1D Kuramoto-Sivashinsky equation
with only partial observations of the environment, thereby answering some
concerns of the community on the efficiency of RL methods in partial
observations situations Recht (2019). In their appendix, Beintema  _et al._
(2020) also confirmed the efficiency of DRL (PPO) to control chaotic systems
(the Lorentz attractor). These two results are therefore encouraging with
regard to the increase and control of turbulences by DRL.

A second milestone to reach is the enlargement of the situations/flows Ren,
Rabault, and Tang (2021), going to three dimensional simulations for instance.
A few works have already been led in three dimensions (see, e.g., Refs. Kühn
_et al._ , 2013a, b; Fan  _et al._ , 2020 for model-free methods, or Ref.
Brackston  _et al._ , 2016 for a model-based application). A pioneering
exploitation of invariances by Belus  _et al._ (2019) could enable DRL
algorithms to deal with an arbitrary large number of actuators while highly
limiting the increase of the training cost, and thereby give access to larger
AFC problems. More generally, the study of symmetries, equivariances and
invariances may enable one to consider larger and more turbulent flow
conditions, but it will require a non-trivial implementation. Indeed, if the
environment holds symmetries, or if the governing equations contain
invariances or equivariances, the ANN will not naturally exploit them. For
instance, with a ’naive’ ANN, one is not sure that the agent will preserve
theoretically-equivariant actions in reality Zeng and Graham (2021).
Nonetheless, with sufficient training data, the ANN may learn invariant and
equivariant properties by setting some constraints between the weights of the
network, but it will then reduce its degrees of freedom Ravanbakhsh,
Schneider, and Póczos (2017); Sannai, Takai, and Cordonnier (2019). Hence, if
one wants a constant number of degrees of freedom and take into account
equivariant and invariant properties, one may require a larger ANN, and thus
more training data. To prevent such an increase, one has to support the ANN to
take advantage of the invariances, equivariances and symmetries when they
exist, by encoding a state or state-action reduced space in which the agent
can work equivalently to the real environment. Zeng and Graham (2021) encoded
a ’symmetry-reductor’ applied to a 1D Kuramoto-Sivashinsky environment
controlled by DDPG. They found robust strategies stabilizing the chaotic
system in the state-action symmetry-reduced subspace, and proved the enhanced
efficiency of DDPG in this subspace in comparison with the ’fully-developed’
space. Their work is an encouraging milestone illustrating how important it
will be to consider equivariances, invariances and symmetries when they exist
in chaotic/turbulent flow control problems, and how to do so. Eventually, by
defining strategies of cooperation between multiple parallel agents, multi-
agent reinforcement learning (MARL) has recently proved its efficiency in AFC
Bae and Koumoutsakos (2022); Guastoni  _et al._ (2023), and may help enhancing
the complexity of the flows considered.

Encouraging avenues for the raise and enlargement of complexity are thus
already under study. In addition, the aforementioned parallelization methods
(see section III.4, and, e.g, Ref. Rabault and Kuhnle, 2019 for a direct
application in AFC) are also promising, as the number of episodes needed for
the training of the NN increases in proportion with the complexity of the
flow. Both invariances and parallel DRL should be considered to deal with
large turbulent flows. Transfer learning Taylor and Stone (2009) is also of
great interest. Multiple studies have already illustrated the effectiveness of
DRL-learned strategies in situations different from the ones considered in the
training phase (see for instance Refs. Tang  _et al._ , 2020; Ren, Rabault,
and Tang, 2021; Ren, Wang, and Tang, 2021), and their robustness when
confronted to an input or output noise (e.g., Refs. Xu and Zhang, 2023; Zeng
and Graham, 2021). This ability the agent has to transfer its learning to a
wider range of flow conditions proves – once more – the robustness of DRL
algorithms and their advantageous properties compared to other AFC methods.
Moreover, a recent work Paris, Beneddine, and Dandois (2023) has shown an
interest in reducing the number of actuators while limiting the deterioration
of the control sequence, exploiting the malleability of NNs. It could thus
enable one to deal with larger problems without increasing the number of
actuators. Eventually, DRL as well as machine learning in general may also
enhance knowledge on physical processes where the classical mechanics
predictions encounter limitations Noack (2019), such as the behavior of
chaotic systems. Taking the example of AlphaGo Silver  _et al._ (2016), not
only did the algorithm impress the community for its performances, but also
because it helped understanding underlying structures in the game, for
instance by choosing strategies commonly thought as inefficient until then.
Therefore, when applied to chaotic systems, DRL methods may highlight unknown
structures and discover new physical properties. The non-opposition-control
solution found by PPO in Ref. Varela  _et al._ , 2022 goes in this direction.
In other words, DRL methods may help enhance our knowledge on physical systems
Rabault and Kuhnle (2022), meanwhile our current knowledge needs to be
exploited to encode better algorithms, taking into account symmetries and
invariances for instance.

Hence, data-driven methods, and more particularly DRL algorithms, have already
demonstrated unrivalled performances in active flow control problems. They now
have to handle larger and more complex situations, as the latter would raise
even more engineering applications and scientific knowledge.

As a last comment, we would emphasize the importance of open-sourcing the
codes of the DRL algorithms applied to flow control problems. As it has been
developed in this review, fruitful methods may emerge from combinations of
existing algorithms. For a reproducibility purpose, and because technical
implementation choices may have a significant influence on the results,
sharing the training data alongside the code and the theoretical details seems
to be essential Rabault and Kuhnle (2022). For examples of shared toolkits,
the reader may refer to Ref. Wang  _et al._ , 2022 and to Ref. Cornejo Maceda,
Lusseyran, and Noack, 2022, which respectively present an open-source Python
platform for DRL and an open-source GP technique for fluid mechanics.

##  V Concluding remarks

In the present review, a general overview of the main RL concepts and of the
DRL framework has first been given. The potential of DRL for the active
control of a flow has then been highlighted – comparatively to either reactive
or predetermined other methods. Not only may the DRL effectiveness be useful
in multiple engineering applications, but it could also help science in
general. Indeed, this exhaustive exploration tool may provide knowledge in
theoretical-physics fields similarly to what it gave to the Go-game-community:
the discovery of previously unknown underlying structures in the studied
system.

For 200 years, science has focused on the application of analytical methods.
This often implies local linearization, as well as stability and modal
analyses. However, in many instances this is not meaningful, because there is
no such thing as a clearly defined base flow that is similar enough to
instantaneous snapshots to perform linearization. Therefore, full-scale
applications may be far enough from a well-defined configuration that
linearization is not representative. Consequently, in general, when handling
non-linear high-dimensional systems, there is no choice but to study directly
the full system. Thus, most analytical tools cannot be used anymore, and this
is one of the reason why methods such as DRL are attractive: they can perform
very well in much more general conditions than traditional methods, at the
cost of being very data intensive. Hence, DRL represents a methodological
shift. It will take time before we as a community understand the full value
and impact of these methods in AFC, and we are currently in the middle of a
massive effort to investigate how far DRL can be taken for AFC applications,
and what new insights can be gained from that.

## Acknowledgements

R.V. acknowledges financial support from ERC grant no. ‘2021-CoG-101043998,
DEEPCONTROL’. Views and opinions expressed are however those of the author(s)
only and do not necessarily reflect those of the European Union or the
European Research Council. Neither the European Union nor the granting
authority can be held responsible for them.

## References

  * Kober, Bagnell, and Peters (2013) J. Kober, J. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” The International Journal of Robotics Research 32, 1238–1274 (2013).
  * Gu  _et al._ (2017) S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in  _2017 IEEE International Conference on Robotics and Automation (ICRA)_ (2017) pp. 3389–3396.
  * OpenAI  _et al._ (2018) OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning Dexterous In-Hand Manipulation,” arXiv e-prints (2018).
  * Li  _et al._ (2016) J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao, “Deep reinforcement learning for dialogue generation,” in  _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ (Association for Computational Linguistics, Austin, Texas, 2016) pp. 1192–1202.
  * Silver  _et al._ (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. baker, M. Lai, A. Bolton, Y. Chen, T. P. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mastering the game of go without human knowledge,” Nature 550, 354–359 (2017).
  * Silver  _et al._ (2016) D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature 529, 484–489 (2016).
  * Brown and Sandholm (2019) N. Brown and T. Sandholm, “Superhuman ai for multiplayer poker,” Science 365, 885 – 890 (2019).
  * Mnih  _et al._ (2013) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” arXiv e-prints (2013).
  * Mnih  _et al._ (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature 518, 529–533 (2015).
  * Szita (2012) I. Szita, “Reinforcement learning in games,” in  _Reinforcement Learning: State-of-the-Art_ , edited by M. Wiering and M. van Otterlo (Springer Berlin Heidelberg, Berlin, Heidelberg, 2012) pp. 539–577.
  * Krizhevsky, Sutskever, and Hinton (2012) A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in  _Advances in Neural Information Processing Systems_ , Vol. 25, edited by F. Pereira, C. Burges, L. Bottou, and K. Weinberger (Curran Associates, Inc., 2012).
  * Hornik, Stinchcombe, and White (1989) K. Hornik, M. B. Stinchcombe, and H. L. White, “Multilayer feedforward networks are universal approximators,” Neural Networks 2, 359–366 (1989).
  * Tesauro (1995) G. Tesauro, “Temporal difference learning and td-gammon,” Commun. ACM 38, 58–68 (1995).
  * Bagnell and Schneider (2001) J. Bagnell and J. Schneider, “Autonomous helicopter control using reinforcement learning policy search methods,” in  _Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164)_ , Vol. 2 (2001) pp. 1615–1620 vol.2.
  * Ng  _et al._ (2004) A. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, and E. Berger, “Inverted autonomous helicopter flight via reinforcement learning,” Proceedings of the International Symposium on Experimental Robotics (2004).
  * Peters and Schaal (2006) J. Peters and S. Schaal, “Policy gradient methods for robotics,” in  _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_ (2006) pp. 2219–2225.
  * Peters and Schaal (2008) J. Peters and S. Schaal, “Learning to control in operational space,” The International Journal of Robotics Research 27, 197–212 (2008).
  * Diuk, Cohen, and Littman (2008) C. Diuk, A. Cohen, and M. L. Littman, “An object-oriented representation for efficient reinforcement learning,” in  _International Conference on Machine Learning_ (2008).
  * Riedmiller  _et al._ (2009) M. Riedmiller, T. Gabel, R. Hafner, and S. Lange, “Reinforcement learning for robot soccer,” Autonomous Robots 27 (2009).
  * Sutton and Barto (2018) R. Sutton and A. Barto,  _Reinforcement learning: an introduction_ (MIT Press, 2018).
  * Brunton and Noack (2015) S. L. Brunton and B. R. Noack, “Closed-Loop Turbulence Control: Progress and Challenges,” Applied Mechanics Reviews 67 (2015).
  * Garnier  _et al._ (2021) P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, and E. Hachem, “A review on deep reinforcement learning for fluid mechanics,” Computers and Fluids 225, 104973 (2021).
  * Vinuesa  _et al._ (2022) R. Vinuesa, O. Lehmkuhl, A. Lozano-Durán, and J. Rabault, “Flow control in wings and discovery of novel approaches via deep reinforcement learning,” Fluids 7, 62 (2022).
  * Pino  _et al._ (2022) F. Pino, L. Schena, J. Rabault, and M. A. Mendez, “Comparative analysis of machine learning methods for active flow control,” arXiv e-prints (2022).
  * Novati, de Laroussilhe, and Koumoutsakos (2021) G. Novati, H. L. de Laroussilhe, and P. Koumoutsakos, “Automating turbulence modelling by multi-agent reinforcement learning,” Nature Machine Intelligence 3, 87–96 (2021).
  * Kurz, Offenhäuser, and Beck (2023) M. Kurz, P. Offenhäuser, and A. Beck, “Deep reinforcement learning for turbulence modeling in large eddy simulations,” International Journal of Heat and Fluid Flow 99, 109094 (2023).
  * Bae and Koumoutsakos (2022) H. J. Bae and P. Koumoutsakos, “Scientific multi-agent reinforcement learning for wall-models of turbulent flows,” Nature Communications 13 (2022).
  * Beck, Flad, and Munz (2019) A. Beck, D. Flad, and C.-D. Munz, “Deep Neural Networks for Data-Driven Turbulence Models,” in  _APS Division of Fluid Dynamics Meeting Abstracts_ , APS Meeting Abstracts (2019) p. G16.006.
  * Beck, Flad, and Munz (2019) A. Beck, D. Flad, and C.-D. Munz, “Deep neural networks for data-driven LES closure models,” Journal of Computational Physics (2019).
  * Beck and Kurz (2021) A. Beck and M. Kurz, “A perspective on machine learning methods in turbulence modeling,” GAMM-Mitteilungen 44 (2021).
  * Kurz  _et al._ (2022a) M. Kurz, P. Offenhäuser, D. Viola, O. Shcherbakov, M. Resch, and A. Beck, “Deep reinforcement learning for computational fluid dynamics on HPC systems,” Journal of Computational Science 65, 101884 (2022a).
  * Kurz  _et al._ (2022b) M. Kurz, P. Offenhaeuser, D. Viola, M. Resch, and A. Beck, “Relexi – A scalable open source reinforcement learning framework for high-performance computing,” Software Impacts 14 (2022b).
  * Lampton, Niksch, and Valasek (2008) A. Lampton, A. Niksch, and J. Valasek,  _Morphing Airfoils with Four Morphing Parameters_ (AIAA Guidance, Navigation and Control Conference and Exhibit, 2008).
  * Lampton, Niksch, and Valasek (2010) A. Lampton, A. Niksch, and J. Valasek, “Reinforcement learning of a morphing airfoil-policy and discrete learning analysis,” Journal of Aerospace Computing, Information, and Communication 7, 241–260 (2010).
  * Ghraieb  _et al._ (2021) H. Ghraieb, J. Viquerat, A. Larcher, P. Meliga, and E. Hachem, “Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows,” Physical Review Fluids 6, 053902 (2021).
  * Batikh, Baldas, and Colin (2017) A. Batikh, L. Baldas, and S. Colin, “Application of active flow control on aircrafts -State of the art,” in  _International Workshop on Aircraft System Technologies_ , Vol. 2017 (Hamburg, Germany, 2017).
  * Noack  _et al._ (2021) B. R. Noack, Y. Li, Z. Yang, G. Y. Cornejo Maceda, F. Lusseyran, P. Oswald, and R. Semann, “Machine learning drag reduction of car and truck models with multiple actuators and sensors,” in  _Aerovehicles 4_ (Berlin, Germany, 2021).
  * Rabault  _et al._ (2019) J. Rabault, M. Kuchta, A. Jensen, U. Réglade, and N. Cerardi, “Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control,” Journal of Fluid Mechanics 865, 281–302 (2019).
  * Tang  _et al._ (2020) H. Tang, J. Rabault, A. Kuhnle, Y. Wang, and T. Wang, “Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning,” Physics of Fluids 32, 053605 (2020).
  * Ren, Rabault, and Tang (2021) F. Ren, J. Rabault, and H. Tang, “Applying deep reinforcement learning to active flow control in weakly turbulent conditions,” Physics of Fluids 33, 037121 (2021).
  * Varela  _et al._ (2022) P. Varela, P. Suárez, F. Alcántara-Ávila, A. Miró, J. Rabault, B. Font, L. M. García-Cuevas, O. Lehmkuhl, and R. Vinuesa, “Deep reinforcement learning for flow control exploits different physics for increasing Reynolds-number regimes,” Actuators 11, 359 (2022).
  * Li and Zhang (2021) J. Li and M. Zhang, “Reinforcement-learning-based control of confined cylinder wakes with stability analyses,” Journal of Fluid Mechanics 932, A44 (2021).
  * Sonoda  _et al._ (2022) T. Sonoda, Z. Liu, T. Itoh, and Y. Hasegawa, “Reinforcement Learning of Control Strategies for Reducing Skin Friction Drag in a Fully Developed Channel Flow,” arXiv e-prints (2022).
  * Guastoni  _et al._ (2023) L. Guastoni, J. Rabault, P. Schlatter, H. Azizpour, and R. Vinuesa, “Deep reinforcement learning for turbulent drag reduction in channel flows,” Preprint arXiv:2301.09889 (2023).
  * Koza (1992) J. R. Koza,  _Genetic programming. On the programming of computers by means of natural selection_ (The MIT Press, 1992).
  * Gautier  _et al._ (2015) N. Gautier, J.-L. Aider, T. Duriez, B. Noack, M. Segond, and M. Abel, “Closed-loop separation control using machine learning,” Journal of Fluid Mechanics 770, 442–457 (2015).
  * Li  _et al._ (2017) R. Li, B. R. Noack, L. Cordier, J. Borée, and F. Harambat, “Drag reduction of a car model by linear genetic programming control,” Experiments in Fluids 58, 103 (2017).
  * Fan, Zhou, and Noack (2019) D. Fan, Y. Zhou, and B. Noack, “Artificial Intelligence Control of Turbulence,” in  _5th Symposium on Fluid Structure-Sound Interactions and Control (FSSIC)_ (Minoa Palace - Resort, Chania, Crete island, Greece, 2019) pp. 1–4.
  * Ren, Wang, and Tang (2019) F. Ren, C. Wang, and H. Tang, “Active control of vortex-induced vibration of a circular cylinder using machine learning,” Physics of Fluids 31, 093601 (2019).
  * Li  _et al._ (2020) H. Li, J. Tan, Z. Gao, and B. R. Noack, “Machine learning open-loop control of a mixing layer,” Physics of Fluids 32, 111701 (2020).
  * Ghraieb  _et al._ (2022) H. Ghraieb, J. Viquerat, A. Larcher, P. Meliga, and E. Hachem, “Single-step deep reinforcement learning for two- and three-dimensional optimal shape design,” AIP Advances 12, 085108 (2022).
  * Viquerat  _et al._ (2020) J. Viquerat, J. Rabault, A. Kuhnle, H. Ghraieb, A. Larcher, and E. Hachem, “Direct shape optimization through deep reinforcement learning,” Journal of Computational Physics 428, 110080 (2020).
  * Gad-el Hak (1996) M. Gad-el Hak, “Modern developments in flow control,” Applied Mechanics Review 49, 365–379 (1996).
  * Scott Collis  _et al._ (2004) S. Scott Collis, R. D. Joslin, A. Seifert, and V. Theofilis, “Issues in active flow control: theory, control, simulation, and experiment,” Progress in Aerospace Sciences 40, 237–289 (2004).
  * Sutton (1988) R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine learning 3, 9–44 (1988).
  * Jaderberg  _et al._ (2017) M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu, “Reinforcement learning with unsupervised auxiliary tasks,” in  _International Conference on Learning Representations_ (2017).
  * Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber, “World Models,” arXiv e-prints , arXiv:1803.10122 (2018).
  * Rabault and Kuhnle (2022) J. Rabault and A. Kuhnle,  _Deep Reinforcement learning applied to Active Flow Control_ (Cambridge University Press, 2022).
  * Barbagallo  _et al._ (2012) A. Barbagallo, G. Dergham, D. Sipp, P. Schmid, and J.-C. Robinet, “Closed-loop control of unsteadiness over a rounded backward-facing step,” Journal of Fluid Mechanics 703, 326–362 (2012).
  * Dahan, Morgans, and Lardeau (2012) J. A. Dahan, A. S. Morgans, and S. Lardeau, “Feedback control for form-drag reduction on a bluff body with a blunt trailing edge,” Journal of Fluid Mechanics 704, 360–387 (2012).
  * Gautier and Aider (2014) N. Gautier and J. L. Aider, “Feed-forward control of a perturbed backward-facing step flow,” Journal of Fluid Mechanics 759, 181 – 196 (2014).
  * Brackston  _et al._ (2016) R. D. Brackston, J. M. García de la Cruz, A. Wynn, G. Rigas, and J. F. Morrison, “Stochastic modelling and feedback control of bistability in a turbulent bluff body wake,” Journal of Fluid Mechanics 802, 726–749 (2016).
  * Kim and Bewley (2007) J. Kim and T. R. Bewley, “A linear systems approach to flow control,” Annual Review of Fluid Mechanics 39, 383–417 (2007).
  * Rigas  _et al._ (2015) G. Rigas, A. S. Morgans, R. D. Brackston, and J. F. Morrison, “Diffusive dynamics and stochastic models of turbulent axisymmetric wakes,” Journal of Fluid Mechanics 778, R2 (2015).
  * Rowley and Dawson (2017) C. W. Rowley and S. T. Dawson, “Model reduction for flow analysis and control,” Annual Review of Fluid Mechanics 49, 387–417 (2017).
  * Queipo  _et al._ (2005) N. V. Queipo, R. T. Haftka, W. Shyy, T. Goel, R. Vaidyanathan, and P. Kevin Tucker, “Surrogate-based analysis and optimization,” Progress in Aerospace Sciences 41, 1–28 (2005).
  * Koziel  _et al._ (2016) S. Koziel, Y. Tesfahunegn, A. Amrit, and L. T. Leifsson, “Rapid multi-objective aerodynamic design using co-kriging and space mapping,” 57th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference (2016).
  * Chua  _et al._ (2018) K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models,” arXiv e-prints , arXiv:1805.12114 (2018).
  * Bellman (1957) R. Bellman, “A markovian decision process,” Journal of Mathematics and Mechanics 6, 679 – 684 (1957).
  * Howard (1960) R. A. Howard,  _Dynamic programming and markov processes_ (Technology Press of Massachusetts Institute of Technology, 1960).
  * Silver  _et al._ (2014) D. Silver, G. Lever, N. M. O. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller, “Deterministic policy gradient algorithms,” in  _International Conference on Machine Learning_ (2014).
  * Watkins (1989) C. Watkins,  _Learning From Delayed Rewards_ , Ph.D. thesis, King’s College, Cambridge, UK (1989).
  * Williams (1992) R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning 8, 229–256 (1992).
  * Meliga, Sipp, and Chomaz (2010) P. Meliga, D. Sipp, and J.-M. Chomaz, “Open-loop control of compressible afterbody flows using adjoint methods,” in  _Seventh IUTAM Symposium on Laminar-Turbulent Transition_ , edited by P. Schlatter and D. S. Henningson (Springer Netherlands, Dordrecht, 2010) pp. 283–288.
  * Shahrabi (2019) A. F. Shahrabi, “The control of flow separation: Study of optimal open loop parameters,” Physics of Fluids 31, 035104 (2019).
  * Degris, White, and Sutton (2012) T. Degris, M. White, and R. S. Sutton, “Linear off-policy actor-critic,” in  _International Conference on Machine Learning_ (2012).
  * Watkins and Dayan (1992) C. Watkins and P. Dayan, “Q-learning,” Machine Learning 8, 279–292 (1992).
  * Sutton and Whitehead (1993) R. S. Sutton and S. D. Whitehead, “Online learning with random representations,” in  _International Conference on Machine Learning_ (1993).
  * Degris, Pilarski, and Sutton (2012) T. Degris, P. M. Pilarski, and R. S. Sutton, “Model-free reinforcement learning with continuous action in practice,” in  _2012 American Control Conference (ACC)_ (2012) pp. 2177–2182.
  * Bellman and Dreyfus (1962) R. Bellman and S. Dreyfus,  _Applied dynamic programming_ (princeton University Press, 1962).
  * Sutton  _et al._ (1999) R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in  _Advances in Neural Information Processing Systems_ , Vol. 12, edited by S. Solla, T. Leen, and K. Müller (MIT Press, 1999).
  * Tesauro and Sejnowski (1989) G. Tesauro and T. J. Sejnowski, “A parallel network that learns to play backgammon,” Artif. Intell. 39, 357–390 (1989).
  * Crites and Barto (1995) R. H. Crites and A. G. Barto, “Improving elevator performance using reinforcement learning,” in  _NIPS_ (1995).
  * Sutton  _et al._ (2000) R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in  _Neural Information Processing Systems_ (2000).
  * Lin (1992) L.-J. Lin, “Self-improving reactive agents based on reinforcement learning, planning and teaching,” Mach. Learn. 8, 293–321 (1992).
  * Riedmiller, Montemerlo, and Dahlkamp (2007) M. Riedmiller, M. Montemerlo, and H. Dahlkamp, “Learning to drive a real car in 20 minutes,” in  _Proceedings of the 2007 Frontiers in the Convergence of Bioscience and Information Technologies_ , FBIT ’07 (IEEE Computer Society, USA, 2007) p. 645–650.
  * LeCun, Bengio, and Hinton (2015) Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521, 436–444 (2015).
  * Schaul  _et al._ (2015) T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized Experience Replay,” arXiv e-prints , arXiv:1511.05952 (2015).
  * Tsitsiklis and Van Roy (1997) J. Tsitsiklis and B. Van Roy, “An analysis of temporal-difference learning with function approximation,” IEEE Transactions on Automatic Control 42, 674–690 (1997).
  * Lillicrap  _et al._ (2015) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv e-prints (2015).
  * Wang  _et al._ (2015) Z. Wang, T. Schaul, M. Hessel, H. V. Hasselt, M. Lanctot, and N. de Freitas, “Dueling network architectures for deep reinforcement learning,” ArXiv abs/1511.06581 (2015).
  * Van Hasselt (2010) H. Van Hasselt, “Double q-learning.” (2010) pp. 2613–2621.
  * van Hasselt, Guez, and Silver (2015) H. van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning with Double Q-learning,” arXiv e-prints (2015).
  * Mnih  _et al._ (2016) V. Mnih, A. Puigdomènech Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous Methods for Deep Reinforcement Learning,” arXiv e-prints (2016).
  * Schulman  _et al._ (2015) J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel, “Trust Region Policy Optimization,” arXiv e-prints (2015).
  * Bucci  _et al._ (2019) M. A. Bucci, O. Semeraro, A. Allauzen, G. Wisniewski, L. Cordier, and L. Mathelin, “Control of chaotic systems by deep reinforcement learning,” Proceedings of the Royal Society of London Series A 475, 20190351 (2019).
  * Schulman  _et al._ (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv e-prints (2017).
  * Kakade and Langford (2002) S. M. Kakade and J. Langford, “Approximately optimal approximate reinforcement learning,” in  _International Conference on Machine Learning_ (2002).
  * Wang  _et al._ (2016) Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas, “Sample Efficient Actor-Critic with Experience Replay,” arXiv e-prints (2016).
  * Szita and Lörincz (2006) I. Szita and A. Lörincz, “Learning tetris using the noisy cross-entropy method,” Neural computation 18, 2936–41 (2006).
  * Peng  _et al._ (2018) X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne, “DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills,” arXiv e-prints , arXiv:1804.02717 (2018).
  * Ren, Wang, and Tang (2021) F. Ren, C. Wang, and H. Tang, “Bluff body uses deep-reinforcement-learning trained active flow control to achieve hydrodynamic stealth,” Physics of Fluids 33, 093602 (2021).
  * Hämäläinen  _et al._ (2018) P. Hämäläinen, A. Babadi, X. Ma, and J. Lehtinen, “PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation,” arXiv e-prints , arXiv:1810.02541 (2018).
  * Abdolmaleki  _et al._ (2018a) A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller, “Maximum a Posteriori Policy Optimisation,” arXiv e-prints (2018a).
  * Abdolmaleki  _et al._ (2018b) A. Abdolmaleki, J. T. Springenberg, J. Degrave, S. Bohez, Y. Tassa, D. Belov, N. Heess, and M. Riedmiller, “Relative Entropy Regularized Policy Iteration,” arXiv e-prints , arXiv:1812.02256 (2018b).
  * Paris, Beneddine, and Dandois (2023) R. Paris, S. Beneddine, and J. Dandois, “Reinforcement-learning-based actuator selection method for active flow control,” Journal of Fluid Mechanics 955 (2023).
  * Simon (1992) H. D. Simon, “Parallel computational fluid dynamics - Implementations and results,” NASA STI/Recon Technical Report A 94, 11384 (1992).
  * Gropp and Smith (1990) W. D. Gropp and E. B. Smith, “Computational fluid dynamics on parallel processors,” Computers and Fluids 18, 289–304 (1990).
  * Gropp  _et al._ (2001) W. D. Gropp, D. K. Kaushik, D. E. Keyes, and B. F. Smith, “High-performance parallel implicit cfd,” Parallel Computing 27, 337–362 (2001), parallel computing in aerospace.
  * Rabault and Kuhnle (2019) J. Rabault and A. Kuhnle, “Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach,” Physics of Fluids 31, 094105 (2019).
  * Nair  _et al._ (2015) A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver, “Massively Parallel Methods for Deep Reinforcement Learning,” arXiv e-prints , arXiv:1507.04296 (2015).
  * Ong, Chavez, and Hong (2015) H. Y. Ong, K. Chavez, and A. Hong, “Distributed deep q-learning,” arXiv (2015).
  * Belus  _et al._ (2019) V. Belus, J. Rabault, J. Viquerat, Z. Che, E. Hachem, and U. Reglade, “Exploiting locality and translational invariance to design effective Deep Reinforcement Learning control of the 1-dimensional unstable falling liquid film,” AIP Advances 9 (2019).
  * Alfredsson and Örlü (2018) P.-H. Alfredsson and R. Örlü, “Large-Eddy BreakUp Devices – A 40 Years Perspective from a Stockholm Horizon,” Flow, Turbulence and Combustion 100 (2018).
  * Bechert and Bartenwerfer (1989) D. W. Bechert and M. Bartenwerfer, “The viscous flow on surfaces with longitudinal ribs,” Journal of Fluid Mechanics 206, 105 – 129 (1989).
  * Greenblatt and Wygnanski (2000) D. Greenblatt and I. J. Wygnanski, “The control of flow separation by periodic excitation,” Progress in Aerospace Sciences 36, 487–545 (2000).
  * Seifert, Darabi, and Wyganski (1996) A. Seifert, A. Darabi, and I. Wyganski, “Delay of airfoil stall by periodic excitation,” Journal of Aircraft - J AIRCRAFT 33, 691–698 (1996).
  * Gad-el Hak, Pollard, and Bonnet (1998) M. Gad-el Hak, A. Pollard, and J. P. Bonnet, “Flow control : fundamentals and practices,” (1998).
  * Moin and Bewley (1994) P. Moin and T. R. Bewley, “Feedback control of turbulence,” Applied Mechanics Reviews 47 (1994).
  * Lumley and Blossey (1998) J. Lumley and P. Blossey, “Control of Turbulence,” Annual Review of Fluid Mechanics 30, 311–327 (1998).
  * Bewley (2001) T. R. Bewley, “Flow control: new challenges for a new renaissance,” Progress in Aerospace Sciences 37, 21–58 (2001).
  * Sutton, Barto, and Williams (1992) R. Sutton, A. Barto, and R. Williams, “Reinforcement learning is direct adaptive optimal control,” IEEE Control Systems Magazine 12, 19–22 (1992).
  * Bersini and Gorrini (1996) H. Bersini and V. Gorrini, “Three connectionist implementations of dynamic programming for optimal control: a preliminary comparative analysis,” in  _Proceedings of International Workshop on Neural Networks for Identification, Control, Robotics and Signal/Image Processing_ (1996) pp. 428–437.
  * Recht (2019) B. Recht, “A tour of reinforcement learning: The view from continuous control,” Annual Review of Control, Robotics, and Autonomous Systems 2, 253–279 (2019).
  * Nian, Liu, and Huang (2020) R. Nian, J. Liu, and B. Huang, “A review on reinforcement learning: Introduction and applications in industrial process control,” Computers and Chemical Engineering 139, 106886 (2020).
  * Kühn  _et al._ (2013a) T. Kühn, V. Ciobaca, R. Rudnik, M. Bauer, B. Gölling, and W. Breitenstein, “Active flow-separation control on a high-lift wing-body configuration,” Journal of Aircraft 50, 56–72 (2013a).
  * Beintema  _et al._ (2020) G. Beintema, A. Corbetta, L. Biferale, and F. Toschi, “Controlling Rayleigh-Bénard convection via Reinforcement Learning,” arXiv e-prints (2020).
  * Jung, Mangiavacchi, and Akhavan (1992) W. Jung, N. Mangiavacchi, and R. Akhavan, “Suppression of turbulence in wall-bounded flows by high-frequency spanwise oscillations,” Physics of Fluids A Fluid Dynamics 4 (1992).
  * Quadrio and Ricco (2004) M. Quadrio and P. Ricco, “Critical assessment of turbulent drag reduction through spanwise wall oscillations,” Journal of Fluid Mechanics 521, 251 – 271 (2004).
  * Atzori  _et al._ (2020) M. Atzori, R. Vinuesa, G. Fahland, A. Stroh, D. Gatti, B. Frohnapfel, and P. Schlatter, “Aerodynamic effects of uniform blowing and suction on a NACA4412 airfoil,” Flow Turbulence and Combustion 105, 735–759 (2020).
  * Atzori  _et al._ (2021) M. Atzori, R. Vinuesa, A. Stroh, D. Gatti, B. Frohnapfel, and P. Schlatter, “Uniform blowing and suction applied to non-uniform adverse-pressure-gradient wing boundary layers,” Physical Review Fluids 6, 113904 (2021).
  * Fahland  _et al._ (2021) G. Fahland, A. Stroh, B. Frohnapfel, M. Atzori, R. Vinuesa, P. Schlatter, and D. Gatti, “Investigation of blowing and suction for turbulent flow control on airfoils,” AIAA Journal 59, 4422–4436 (2021).
  * Laadhari, Skandaji, and Morel (1994) F. Laadhari, L. Skandaji, and R. Morel, “Turbulence reduction in a boundary layer by a local spanwise oscillating surface,” Physics of Fluids 6 (1994).
  * Choi (2002) K.-S. Choi, “Near-wall structure of turbulent boundary layer with spanwise-wall oscillation,” Physics of Fluids 14 (2002).
  * Karniadakis and Choi (2003) G. Karniadakis and K.-S. Choi, “Mechanisms on transverse motions in turbulent wall flows,” Annual Review of Fluid Mechanics - ANNU REV FLUID MECH 35, 45–62 (2003).
  * Min  _et al._ (2006) T. Min , S. Kang , J. Speyer, and J. Kim, “Sustained sub-laminar drag in a fully developed channel flow,” Journal of Fluid Mechanics 558, 309 – 318 (2006).
  * Lieu, Moarref, and Jovanović (2010) B. K. Lieu, R. Moarref, and M. R. Jovanović, “Controlling the onset of turbulence by streamwise travelling waves. Part 2. Direct numerical simulation,” Journal of Fluid Mechanics 663, 100–119 (2010).
  * Kametani and Fukagata (2011) Y. Kametani and K. Fukagata, “Direct numerical simulation of spatially developing turbulent boundary layers with uniform blowing or suction,” Journal of Fluid Mechanics 681, 154 – 172 (2011).
  * Kühn  _et al._ (2013b) T. Kühn, V. Ciobaca, R. Rudnik, M. Bauer, B. Gölling, and W. Breitenstein, “Active flow-separation control on a high-lift wing-body configuration,” Journal of Aircraft 50, 56–72 (2013b).
  * Choi, Moin, and Kim (1994) H. Choi, P. Moin, and J. Kim, “Active turbulence control for drag reduction in wall-bounded flows,” Journal of Fluid Mechanics 262, 75 – 110 (1994).
  * Sargent (2000) R. Sargent, “Optimal control,” Journal of Computational and Applied Mathematics 124, 361–371 (2000), numerical Analysis 2000. Vol. IV: Optimization and Nonlinear Equations.
  * Cantwell (1981) B. J. Cantwell, “Organized motion in turbulent flow,” Annual Review of Fluid Mechanics 13, 457–515 (1981).
  * Robinson (1991) S. K. Robinson, “Coherent motions in the turbulent boundary layer,” Annual Review of Fluid Mechanics 23, 601–639 (1991).
  * Hammond, Bewley, and Moin (1998) E. P. Hammond, T. R. Bewley, and P. Moin, “Observed mechanisms for turbulence attenuation and enhancement in opposition-controlled wall-bounded flows,” Physics of Fluids 10, 2421–2423 (1998).
  * Chung and Tariq (2011) Y. Chung and T. Tariq, “Effectiveness of active flow control for turbulent skin friction drag reduction,” Physics of Fluids - PHYS FLUIDS 23 (2011).
  * Stroh  _et al._ (2015) A. Stroh, B. Frohnapfel, P. Schlatter, and Y. Hasegawa, “A comparison of opposition control in turbulent boundary layer and turbulent channel flow,” Physics of Fluids 27, 075101 (2015).
  * Bewley, Moin, and Temam (2001) T. R. Bewley, P. Moin, and R. Temam, “Dns-based predictive control of turbulence: An optimal benchmark for feedback algorithms,” Journal of Fluid Mechanics 447, 179 – 225 (2001).
  * Lee, Kim, and Choi (1998) C. Lee, J. Kim, and H. Choi, “Suboptimal control of turbulent channel flow for drag reduction,” Journal of Fluid Mechanics 358, 245–258 (1998).
  * Hasegawa and Kasagi (2011) Y. Hasegawa and N. Kasagi, “Dissimilar control of momentum and heat transfer in a fully developed turbulent channel flow,” Journal of Fluid Mechanics 683, 57–93 (2011).
  * Wang, Hu, and Blonigan (2014) Q. Wang, R. Hu, and P. Blonigan, “Least squares shadowing sensitivity analysis of chaotic limit cycle oscillations,” Journal of Computational Physics 267, 210–224 (2014).
  * Blanchard  _et al._ (2021) A. B. Blanchard, G. Y. Cornejo Maceda, D. Fan, Y. Li, Y. Zhou, B. R. Noack, and T. P. Sapsis, “Bayesian optimization for active flow control,” Acta Mechanica Sinica 37, 1786–1798 (2021).
  * Noack (2019) B. R. Noack, “Closed-Loop Turbulence Control-From Human to Machine Learning (and Retour),” in  _Proceedings of the 4th Symposium on Fluid Structure-Sound Interactions and Control (FSSIC)_ , edited by Zhou, Y., Kimura, M., Peng, G. Lucey, A.D., Huang, and L. (Springer, 2019) pp. 23–32.
  * Tokarev, Palkin, and Mullyadzhanov (2020) M. Tokarev, E. Palkin, and R. Mullyadzhanov, “Deep reinforcement learning control of cylinder flow using rotary oscillations at low reynolds number,” Energies 13 (2020).
  * Flinois and Colonius (2015) T. L. B. Flinois and T. Colonius, “Optimal control of circular cylinder wakes using long control horizons,” Physics of Fluids 27, 087105 (2015).
  * Li  _et al._ (2019) Y. Li, W. Cui, Q. Jia, Q. Li, Z. Yang, M. Morzyński, and B. R. Noack, “Explorative gradient method for active drag reduction of the fluidic pinball and slanted Ahmed body,” arXiv e-prints , arXiv:1905.12036 (2019).
  * Mendez  _et al._ (2023) M. A. M. Mendez, A. Ianiro, B. R. Noack, and S. L. Brunton,  _Data-Driven Fluid Mechanics: Combining First Principles and Machine Learning_ (Cambridge University Press, 2023).
  * Qin  _et al._ (2021) S. Qin, S. Wang, J. Rabault, and G. Sun, “An application of data driven reward of deep reinforcement learning by dynamic mode decomposition in active flow control,” arXiv e-prints , arXiv:2106.06176 (2021).
  * Castellanos  _et al._ (2021) R. Castellanos, I. de La Fuente, G. Cornejo Maceda, B. Noack, A. Ianiro, and S. Discetti, “Machine Learning flow control in the few sensors limit,” in  _APS Division of Fluid Dynamics Meeting Abstracts_ , APS Meeting Abstracts (2021) p. H23.008.
  * Gazzola  _et al._ (2016) M. Gazzola, A. Tchieu, D. Alexeev, A. de Brauer, and P. Koumoutsakos, “Learning to school in the presence of hydrodynamic interactions,” Journal of Fluid Mechanics 789, 726–749 (2016).
  * Novati  _et al._ (2017) G. Novati, S. Verma, D. Alexeev, D. Rossinelli, W. van Rees, and P. Koumoutsakos, “Synchronisation through learning for two self-propelled swimmers,” Bioinspiration and Biomimetics 12, 036001 (2017).
  * Verma, Novati, and Koumoutsakos (2018) S. Verma, G. Novati, and P. Koumoutsakos, “Efficient collective swimming by harnessing vortices through deep reinforcement learning,” Proceedings of the National Academy of Sciences 115, 5849–5854 (2018).
  * Whittlesey, Liska, and Dabiri (2010) R. W. Whittlesey, S. Liska, and J. O. Dabiri, “Fish schooling as a basis for vertical axis wind turbine farm design,” Bioinspiration and Biomimetics 5, 035005 (2010).
  * Rabault  _et al._ (2020) J. Rabault, F. Ren, W. Zhang, H. Tang, and H. Xu, “Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization,” Journal of Hydrodynamics 32, 234–246 (2020).
  * Vinuesa and Brunton (2022) R. Vinuesa and S. Brunton, “Enhancing computational fluid dynamics with machine learning,” Nature Computational Science 2, 358–366 (2022).
  * Brunton, Noack, and Koumoutsakos (2020) S. L. Brunton, B. R. Noack, and P. Koumoutsakos, “Machine learning for fluid mechanics,” Annual Review of Fluid Mechanics 52, 477–508 (2020).
  * Fan  _et al._ (2020) D. Fan, L. Yang, Z. Wang, M. S. Triantafyllou, and G. E. Karniadakis, “Reinforcement learning for bluff body active flow control in experiments and simulations,” Proceedings of the National Academy of Sciences 117, 26091–26098 (2020).
  * Xu  _et al._ (2020) H. Xu, W. Zhang, J. Deng, and J. Rabault, “Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning,” Journal of Hydrodynamics 32, 254–258 (2020).
  * Jiao  _et al._ (2021) Y. Jiao, F. Ling, S. Heydari, N. Heess, J. Merel, and E. Kanso, “Learning to swim in potential flow,” Phys. Rev. Fluids 6, 050505 (2021).
  * Yan  _et al._ (2019) X. Yan, J. Zhu, M. Kuang, and X. Wang, “Aerodynamic shape optimization using a novel optimizer based on machine learning techniques,” Aerospace Science and Technology 86, 826–835 (2019).
  * Yonekura and Hattori (2019) K. Yonekura and H. Hattori, “Framework for design optimization using deep reinforcement learning,” Structural and Multidisciplinary Optimization 60 (2019).
  * Xu and Zhang (2023) D. Xu and M. Zhang, “Reinforcement-learning-based control of convectively unstable flows,” Journal of Fluid Mechanics 954, A37 (2023).
  * Schäfer  _et al._ (1996) M. Schäfer, S. Turek, F. Durst, E. Krause, and R. Rannacher, “Benchmark computations of laminar flow around a cylinder,” in  _Flow Simulation with High-Performance Computers II: DFG Priority Research Programme Results 1993–1995_ , edited by E. H. Hirschel (Vieweg+Teubner Verlag, 1996) pp. 547–566.
  * Shimomura  _et al._ (2020) S. Shimomura, S. Sekimoto, A. Oyama, K. Fujii, and H. Nishida, “Closed-loop flow separation control using the deep q network over airfoil,” AIAA Journal 58, 1–11 (2020).
  * Xu, Lombard, and Sherwin (2017) H. Xu, J.-E. W. Lombard, and S. J. Sherwin, “Influence of localised smooth steps on the instability of a boundary layer,” Journal of Fluid Mechanics 817, 138–170 (2017).
  * Xu  _et al._ (2017) H. Xu, S. M. Mughal, E. R. Gowree, C. J. Atkin, and S. J. Sherwin, “Destabilisation and modification of tollmien–schlichting disturbances by a three-dimensional surface indentation,” Journal of Fluid Mechanics 819, 592–620 (2017).
  * Yu  _et al._ (2021) J. Yu, D. Fan, B. R. Noack, and Y. Zhou, “Genetic-algorithm-based artificial intelligence control of a turbulent boundary layer,” Acta Mechanica Sinica 37, 1739–1747 (2021).
  * Zeng and Graham (2021) K. Zeng and M. Graham, “Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics,” Physical Review E 104 (2021).
  * Ravanbakhsh, Schneider, and Póczos (2017) S. Ravanbakhsh, J. Schneider, and B. Póczos, “Equivariance through parameter-sharing,” in  _Proceedings of the 34th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, Vol. 70, edited by D. Precup and Y. W. Teh (PMLR, 2017) pp. 2892–2901.
  * Sannai, Takai, and Cordonnier (2019) A. Sannai, Y. Takai, and M. Cordonnier, “Universal approximations of permutation invariant/equivariant functions by deep neural networks,” arXiv e-prints , arXiv:1903.01939 (2019).
  * Taylor and Stone (2009) M. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” Journal of Machine Learning Research 10, 1633–1685 (2009).
  * Wang  _et al._ (2022) Q. Wang, L. Yan, G. Hu, C. Li, Y. Xiao, H. Xiong, J. Rabault, and B. R. Noack, “Drlinfluids: An open-source python platform of coupling deep reinforcement learning and openfoam,” Physics of Fluids 34, 081801 (2022).
  * Cornejo Maceda, Lusseyran, and Noack (2022) G. Y. Cornejo Maceda, F. Lusseyran, and B. R. Noack, “xMLC – A Toolkit for Machine Learning Control,” arXiv e-prints , arXiv:2208.13172 (2022).