# Topological phase transition in the periodically forced Kuramoto Model

E. A. P. Wright<EMAIL_ADDRESS>S. Yoon  J. F. F. Mendes  A. V. Goltsev
Department of Physics $\&$ I3N, University of Aveiro, 3810-193 Aveiro,
Portugal  A.F. Ioffe Physico-Technical Institute, 194021 St. Petersburg,
Russia

###### Abstract

A complete bifurcation analysis of explicit dynamical equations for the
periodically forced Kuramoto model was performed in [L. M. Childs and S. H.
Strogatz. Chaos 18, 043128 (2008)], identifying all bifurcations within the
model. We show that the phase diagram predicted by this analysis is
incomplete. Our numerical analysis of the equations reveals that the model can
also undergo an abrupt phase transition from oscillations to wobbly rotations
of the order parameter under increasing field frequency or decreasing field
strength. This transition was not revealed by bifurcation analysis because it
is not caused by a bifurcation, and can neither be classified as first nor
second order since it does not display critical phenomena characteristic of
either transition. We discuss the topological origin of this transition and
show that it is determined by a singular point in the order-parameter space.

###### keywords:

Synchronization , Topological transition , Winding number , Bifurcation
analysis , Singular point , Entrainment

††journal: Chaos, Solitons & Fractals

##  1  Introduction

The Kuramoto model describes synchronization between coupled phase
oscillators, and has been studied extensively since it was first introduced
[1]. This representative model has attracted much attention for its wide range
of applications in physics, neuroscience, biology, and the social sciences as
well as the variety of collective dynamical phenomena it demonstrates [2, 3,
4, 5], including exotic collective states such as chimera states [6] and
Bellerophon states [7], well known first and second order phase transitions
[2, 3, 4, 5], and an unusual hybrid phase transition [8]. Here, from among the
different lines of enquiry, we consider the problem of entrainment by an
external periodic field. The phase diagram for a system of uniformly coupled
Kuramoto oscillators in an external periodic field was first reported in [9].
More recently, Childs and Strogatz performed a complete bifurcation analysis
of explicit dynamical equations for the periodically forced Kuramoto model,
revealing the exact locations of all bifurcations [10].

In this work, we report a new phase transition in the periodically forced
Kuramoto model, based on numerical and analytical analysis of the explicit
dynamical equations derived in [10]. We show that this transition cannot be
revealed by bifurcation analysis because it is not caused by a bifurcation.
Moreover, it cannot be classified as a first or second order phase transition
since it does not display critical phenomena characteristic of either
transition. Our analysis reveals the topological character of this transition,
and the crucial role of a singular point in the order-parameter space.
Topological phase transitions have been observed and studied in quantum
systems such as fractional quantum Hall liquids and topological isolators [11,
12, 13]. Our work shows that a topological transition can also occur in a
classical system such as the forced Kuramoto model.

In the next section, we provide an overview of the forced Kuramoto model, the
order-parameter space, and a brief summary of the physical behavior displayed
by the model. In Section 3, we present an updated phase diagram and our
analysis of the transition. Finally, we discuss our findings in Section 4.

##  2  The Forced Kuramoto Model

In the Kuramoto model with a periodic field of strength $F$ and angular
frequency $\sigma$, the phase $\theta_{i}$ of the $i^{\text{th}}$ phase
oscillator is given by

$\frac{d\theta_{i}}{dt}=\omega_{i}+\frac{K}{N}\sum_{j=1}^{N}\sin(\theta_{j}-\theta_{i})+F\sin(\sigma
t-\theta_{i}),$ (1)

where $\omega_{i}$ is the oscillator’s natural frequency, $K$ is the uniform
coupling strength, and $N$ the number of oscillators in the system. The
macroscopic state of the system is conveniently described by the complex order
parameter

$z=\rho e^{i\psi}=\frac{1}{N}\sum_{j=1}^{N}e^{i\theta_{j}},$ (2)

where the parameter $\rho$ characterizes the degree of synchronization between
the oscillators ($0\leq\rho\leq 1$) and the phase $\psi$ shows the direction
of alignment. Applying the Ott-Antonsen ansatz [14] to a system of oscillators
with natural frequencies drawn from a Lorentzian distribution of spread
$\Delta=1$, in a frame rotating at the field frequency $\sigma$, the
macroscopic dynamics of the system are reduced to

$\displaystyle\frac{d\rho}{dt}$
$\displaystyle=-\rho+\frac{K}{2}\rho(1{-}\rho^{2}){+}\frac{F}{2}(1{-}\rho^{2})\cos\psi,$
(3) $\displaystyle\frac{d\psi}{dt}$
$\displaystyle=-\Omega-\frac{F}{2}\frac{(1+\rho^{2})}{\rho}\sin\psi,$ (4)

where $\Omega=\sigma-\omega_{0}$ is the detuning parameter, and $\omega_{0}$
is the average natural frequency of oscillators. Note that the resulting
equations are symmetric with respect to the replacement
$\Omega\rightarrow-\Omega$ and $\psi\rightarrow-\psi$. For simplicity, we will
assume $\omega_{0}=0$, so that $\Omega=\sigma$. Equivalently, one may consider
a frame rotating with frequency $\omega_{0}$, where the field frequency is
$\sigma-\omega_{0}$.

Eqs. (3) and (4) describe the motion of the system in a two-dimensional order-
parameter space $(\text{Re}(z),\text{Im}(z))=(\rho\cos\psi,\rho\sin\psi)$
where positive rotation of $\psi$ is counterclockwise. It is important to note
that the state $z=0$ is a singular point in this space because $\psi$ becomes
undefined when $\rho=0$. Since any trajectory through this order-parameter
space requires the function $\psi(t)$ to be analytic at every time $t$, e.g.
the first derivative defines angular velocity, any trajectory through the
singular point $z=0$ is forbidden.

The complete bifurcation analysis of Eqs. $(\ref{eq:rho})$ and
$(\ref{eq:psi})$ was performed by Childs and Strogatz in [10] where one can
find the explicit stability diagram, all bifurcations curves, and phase
portraits of dynamical states of the forced Kuramoto model at different model
parameters $F,\Omega$ and $K$. In short, the model demonstrates the following
physical behavior. First, in the absence of the field, when the coupling $K$
between phase oscillators is larger than the critical coupling $K_{c}=2$,
interaction results in a synchronous rotation of phase oscillators with the
group velocity $\omega_{0}$ and order parameter $\rho=\sqrt{(K-2)/K}$. At
$K=K_{c}$, the rotational symmetry is spontaneously broken and the phases
$\theta_{i}$ of a finite fraction of oscillators become aligned along an
arbitrary direction $\psi$. Second, upon applying a field with strength $F$ at
detuning $|\Omega|$ sufficiently smaller than $F$, the synchronized group
becomes entrained by the field, for a broad range of model parameters (see
phase I in Fig. 1). In the original non-rotating frame, the entrained state is
phase- and frequency-locked to the field and a finite fraction of phase
oscillators rotates synchronously with the field frequency $\sigma$. If
$\Omega>0$, the order parameter $z$ lags behind the field, i.e., the phase
$\psi$ is negative and the order parameter $z$ lies in the lower half-plane of
the order-parameter space. If $\Omega<0$, then the order parameter $z$ is
ahead of the field, the phase $\psi$ is positive and $z$ lies in the upper
half-plane. The latter symmetry follows from the symmetry of Eqs.
$(\ref{eq:rho})$ and $(\ref{eq:psi})$ with respect to the replacement
$\Omega\rightarrow-\Omega$ and $\psi\rightarrow-\psi$. Third, increasing
$\Omega$ at fixed $F$ disrupts the entrained state and leads to periodic
dynamics in the rotating frame, as the system undergoes a SNIPER, saddle-node,
or Hopf bifurcation [10]. As discussed below, we found that when entrainment
is disrupted, coupled oscillators can either oscillate synchronously with
respect to the field direction (phase II in Fig. 1) or drift at a frequency
other than the field frequency (phase III in Fig. 1).

##  3 SPOR transition

Our numerical analysis of Eqs. $(\ref{eq:rho})$ and $(\ref{eq:psi})$ revealed
that the phase diagram in [10] is incomplete since it misses a phase
transition from the phase with oscillations (phase II in Fig. 1) into a phase
where the phase oscillators demonstrate a wobbling rotation around the
singular point $z=0$ with an angular frequency that differs from the field
frequency (phase III in Fig. 1). In order to understand the origin and
properties of this transition, which we have tentatively named a Singular
Point Oscillation-to-Rotation (SPOR) transition, we performed a detailed
numerical analysis of Eqs. $(\ref{eq:rho})$ and $(\ref{eq:psi})$.

Figure 1:  Partial phase diagram for the Kuramoto model in a homogenous field
with strength $F$ and detuning $\Omega$ at $K=5$. Each phase is characterized
by distinct dynamics of $\psi(t)$. Phase $\mathrm{I}$ corresponds to the
entrained phase. Phase $\mathrm{II}$ is the oscillating phase. Phase
$\mathrm{III}$, is the rotating phase. Full circles (blue) define a boundary
(the SPOR transition) separating phases $\mathrm{II}$ (shadowed region) and
$\mathrm{III}$. The upper boundary of phase $\mathrm{I}$ was calculated
following [10]: the dash-dotted line (black) indicates a SNIPER bifurcation,
which becomes a Saddle-Node bifurcation (full black line) at the triangular
marker, which in turn becomes a Hopf bifurcation (dotted green line) at the
square marker. A detailed phase diagram in the region I is presented in [10].

First, we studied the oscillations in phase II, by fixing the field strength
$F$ and increasing $\Omega$. The non-uniform dynamics of $z$ in phase II are
typified in Fig. 2(b), for oscillations born from a Hopf bifurcation. In this
phase, the order parameter $z$ describes a limit cycle in the lower half plane
of the order-parameter space. The amplitude of the oscillations in $\psi$ is
bound between $\psi_{max}$ and $\psi_{min}$, so that
$\psi_{max}-\psi_{min}<\pi$. As the detuning $\Omega$ increases, the amplitude
$\psi_{max}-\psi_{min}$ tends to $\pi$ at a critical detuning $\Omega_{C}$,
which determines the boundary between phases II and III. Fig. 2(b) shows that
over one period of oscillation, the order parameter slowly falls behind the
field ($\psi$ moves slowly from 0 to $-\pi$) and then quickly catches up
($\psi$ quickly moves from $-\pi$ to 0). The sharp increase of $\psi$ from a
value a little bit above $-\pi$ to a value a little bit below 0 corresponds to
fast motion of $z$ along the upper part of the limit cycle, due to high
angular velocity $d\psi/dt$ caused by the $1/\rho$ singularity in Eq.
$(\ref{eq:psi})$. Finally, we note that in the rotating frame the angular
velocity $d\psi/dt$ averaged over the period of oscillations is zero. Thus,
$z$ is on average frequency-locked to the field. At $\Omega>\Omega_{C}$, we
enter phase III, and the order parameter $z$ starts rotating clockwise in the
rotating frame, around the singular point $z=0$, as shown in Figs. 2(a) and
2(c). Note that the parameter $\rho$ has the same behavior above and below
$\Omega_{C}$ (see Figs. 2(b) and 2(c)).

Figure 2: (a) Orbits of the order parameter $z$ at different $\Omega$ and
parameters $F=3.5$ and $K=5$. Dynamics of $\psi$ and $\rho$ in (b) the
oscillating phase at $\Omega=3.90754900$ below the SPOR transition, and (c)
the rotating phase at $\Omega=3.90754901$ above the SPOR transition. Our
estimation of the critical detuning parameter is
$3.90754900<\Omega_{C}<3.90754901$. The origin of the complex plane (singular
point) is marked with a star. The remaining markers indicate the unstable
fixed point (spiral) and are mapped to each orbit by color. The inset in (a)
shows the region of the order parameter space near the singular point, and a
clear change in the position of the orbits relative to the singular point for
a small change ($10^{-8}$) in $\Omega$. Arrowheads in (a) indicate the
direction of motion along the orbits.

In order to understand the origin of the SPOR transition, we analyzed the
behavior of the bifurcation point and limit cycle. In phases II (oscillating
phase) and III (rotating phase) of the phase diagram there is only one
bifurcation point, an unstable spiral. In the oscillating phase, the limit
cycle lies in the lower half-plane as shown in Fig. 2(a). Increasing $\Omega$
moves the system away from the critical boundary with region I, expanding the
limit cycle around the bifurcation point, and moving the upper part of the
limit cycle towards the singular point $z=0$. For $\Omega<\Omega_{C}$, the
limit cycle remains in the lower half-plane, never crossing the singular
point. Increasing $\Omega$ from below to above $\Omega_{C}$ causes the
bifurcation point to move continuously towards the singular point $z=0$, as
depicted in Fig. 2(a). Next, we numerically analyzed the shape and curvature
of the limit cycles and found no peculiarities even at $\Omega$ very close
$(10^{-8}$) to $\Omega_{C}$. We used the well known equation for the curvature
$\kappa$ of a curve, given by a function $\rho=\rho(\psi)$ in polar
coordinates $(\rho,\psi)$:

$\kappa(\psi)=\frac{|\rho^{2}+2(\rho^{\prime})^{2}-\rho\rho^{\prime\prime}|}{[\rho^{2}+(\rho^{\prime})^{2}]^{3/2}},$
(5)

where $\rho^{\prime}\equiv d\rho/d\psi$. We found the function $\rho(\psi)$,
which defines the limit cycle, and its derivatives by use of the numerical
solution $\rho(t)$ and $\psi(t)$. In the tested range of $\Omega$ below
$\Omega_{C}$, the limit cycles presented no peculiarity in shape, such as
flattening (which corresponds to $\kappa\rightarrow 0$). The curvature is
finite and nonzero at all points on the limit cycle, as shown in Fig. 3 for
the limit cycle of oscillations at $\Omega$ very close to $\Omega_{C}$.

Figure 3:  Curvature $\kappa$ versus $\psi$ of the limit cycle of oscillations
near the critical point of the transition to rotation. Parameters: $K=5$,
$F=3.5$, $\Omega=3.907549$.

We also extended our numerical analysis to limit cycles above the critical
point $\Omega_{C}$. At $\Omega>\Omega_{C}$, the singular point is encircled by
the limit cycle, as shown in Fig. 2(a). The shape of the limit cycle, its
curvature, and the tangential velocity are similar to those below
$\Omega_{C}$. Although we found the angular velocity diverges in the vicinity
of the singular point $z=0$ as $\Omega$ tends to $\Omega_{C}$, namely that
$\max(d\psi/dt)\rightarrow+\infty$ from below and
$\min(d\psi/dt)\rightarrow-\infty$ from above, the tangential velocity
$(dx/dt,dy/dt)$ is finite at all points on the limit cycle, where
$x=\text{Re}(z)$ and $y=\text{Im}(z)$. Finally, we analyzed the relaxation
time above and below $\Omega_{C}$ and found that the relaxation time is finite
and does not demonstrate critical behavior such as critical slowing down.
Likewise, we found no evidence of hysteresis depending on initial conditions.
Thus, we found no evidence suggesting the observed transition is second- or
first-order, as would otherwise be suggested by the presence of the above-
mentioned phenomena.

In the original non-rotating frame, the phase of the order parameter $z$
equals $\widetilde{\psi}=\sigma t+\psi$. Therefore, the average cycling
frequency $f$ of $z$ over one period $T$ of oscillations or wobbling rotations
is

$f=\frac{1}{2\pi
T}\int_{t}^{t+T}dt^{\prime}\frac{d\widetilde{\psi}}{dt^{\prime}}=\frac{\sigma}{2\pi}+\frac{1}{2\pi
T}\left[\psi(t+T)-\psi(t)\right],$ (6)

where $t$ is an arbitrary point in time, and the period $T$ is a function of
$\Omega$, $F$, and $K$. From Eq. (6), it follows that $f=\sigma/(2\pi)$ in the
oscillating phase, where $\psi(t+T)=\psi(t)$, showing the system is on average
frequency-locked to the field. In the rotating phase, the average cycling
frequency is $f=\sigma/(2\pi)-1/T$ since $\psi(t+T)=\psi(t)-2\pi$ for
clockwise rotation, and the frequency $f$ tends to $\omega_{0}$ for increasing
$\Omega$ since a high frequency field does not impact coupled phase
oscillators. Thus, the SPOR transition from oscillations to wobbling rotations
appears as an abrupt drop in the average cycling frequency $f$ at
$\Omega_{C}$, and the cycling frequency $f$ decreases for $\Omega>\Omega_{C}$,
as shown in Fig. 4 (c). The drop equals the inverse of the oscillation period
$T$ at the critical point. Our numerical results allow us to conclude that the
only important difference between limit cycles below and above $\Omega_{C}$ is
in the relative position of the singular point $z=0$. The singular point lies
outside the limit cycles at $\Omega<\Omega_{C}$ and is encircled by the limit
cycles at $\Omega>\Omega_{C}$, as shown in Fig. 2(a). As a result, these two
types of limit cycles, each associated with distinct dynamical states, acquire
different topological properties. Based on the topological theory of ordered
media [15], we can characterize limit cycles by a winding number $n$. The
field $\textbf{s}(\textbf{r})=(\rho\cos\psi,\rho\sin\psi)$ is known on all
points r on a limit cycle [see, for example, vectors A and B in Fig. 4(d)]. We
can measure the total angle $\psi$ when the vector $\textbf{s}(\textbf{r})$
turns as r traverses the complete limit cycle (counterclockwise increments in
angle are positive and clockwise increments are negative). Since
$\textbf{s}(\textbf{r})$ is continuous on the cycle, this angle must be an
integral multiple of $2\pi$.

Figure 4: (a) Minimum value of the amplitude $\rho\left(t\right)$ of the
complex order parameter $z=\rho\exp(i\psi)$ as a function of the detuning
$\Omega$. (b) Winding number $n$, Eq. 7, of the orbit described by $z$ in a
frame rotating at the field frequency, for both positive and negative
$\Omega$. (c) Average cycling frequency of rotation $f$ in the original non-
rotating frame, for both positive and negative $\Omega$. The dashed line
indicates non-oscillating phases, and $\Omega_{C}$ is the critical detuning.
(d) Schematic representation of two orbits for detuning $\Omega$ slightly
greater (dashed line) and slightly smaller (solid line) than $\Omega_{C}$.
These orbits differ simply by the translation represented by the green dashed
vector connecting unstable spiral points 1 and 2, and the translation is
exaggerated for illustrative purposes. The tangential velocity at equivalent
points under the translation is the same for both orbits, as shown at points A
and B. The red vectors from the origin of the complex plane (marked with a
star) to points A and B describe orbits with winding numbers 0 (solid vector)
and $-1$ (dashed vector).

In addition, since we are mapping the temporal behavior of the order parameter
$z(t)$ to the order-parameter space $(\rho,\psi)$, we parameterize the cycle
by time $t$. The winding number $n$ of a limit cycle may then be defined as

$n=\frac{1}{2\pi}\int_{t}^{t+T}dt^{\prime}\frac{d\psi}{dt^{\prime}}=\frac{1}{2\pi}[\psi(t+T)-\psi(t)],$
(7)

where $T$ is the period and $t$ is an arbitrary point in time. In the case of
oscillations, the winding number of the limit cycle is $n=0$, since
$\psi(t+T)=\psi(t)$. For positive $\Omega>\Omega_{C}$ (clockwise rotation),
rotations are characterized by $n=-1$, since $\psi(t+T)=\psi(t)-2\pi$.
Moreover, by comparing Eqs. (7) and (6), we can immediately see that the
abrupt drop in cycling frequency $f$ measured in the original non-rotating
frame is directly related with the winding number $n$ as follows

$f=\frac{\sigma}{2\pi}+\frac{n}{T}.$ (8)

In summary, our analysis revealed a topological phase transition (the SPOR
transition) in the forced Kuramoto model, from oscillations to wobbly
rotations of the complex order parameter $z$. The transition takes place at
$\Omega=\Omega_{C}$, as the orbit of $z(\rho,\psi)$ in the rotating frame
approaches the singular point $z=0$, such that $\mathrm{min}(\rho(t))\to 0$ as
shown in Fig. 4(a). In the non-rotating frame, the transition is marked by an
abrupt drop in the average cycling frequency $f$, as shown in Fig. 4(c).
Numerical analysis of the curvature shows the limit cycles immediately below
and above $\Omega_{C}$ are identical, and related by a simple translation in
the order-parameter space, as depicted schematically in Fig. 4(d). However,
the existence of the singular point $z=0$ makes this translation impossible,
since an adiabatic translation would require the limit cycle to pass through
the singular point, which is forbidden (as explained in Sec. 2). In addition,
the oscillating and rotating phases can be distinguished by a topological
characteristic, the winding number $n$ of the limit cycles, as shown in Fig.
4(c). Notably, the average cycling frequency $f$ measured in the non-rotating
frame is directly related with the winding number.

##  4 Discussion

In this paper we found that, by increasing the frequency or decreasing the
strength of an external field, the periodically forced Kuramoto model
undergoes an abrupt phase transition from a phase with oscillations to a phase
with wobbly rotations of the order parameter. We call this a ’Singular Point
Oscillation-to-Rotation’ (SPOR) phase transition. In the original non-rotating
frame, the SPOR transition appears as an abrupt drop in average group
frequency of phase oscillators.

Our analysis of the dynamical behavior of the forced Kuramoto model shows that
the SPOR transition can neither be classified as a first nor as a second order
phase transition since it does not display critical phenomena characteristic
of either transition. First, according to the classification of phase
transitions in Landau’s phenomenological theory [16], second order
(continuous) phase transitions are accompanied by symmetry breaking across the
transition and a gradual increase of the order parameter in the ordered phase.
In the case of the SPOR transition, symmetry breaking is absent. Second, the
proximity to the critical point of a continuous transition is signalled by an
increase of critical fluctuations that result in critical phenomena, such as a
strong increase in susceptibility, correlation length, relaxation rate (known
as critical slowing down of dynamics), etc. Our detailed analysis of the
dynamical behavior of the forced Kuramoto model near the SPOR transition
revealed no anomaly in behavior in the corresponding fixed point, the
curvature of the limit cycle, and the relaxation rate. Third, first order
(discontinuous) phase transitions are characterized by an abrupt appearance of
order, hysteresis, and a region of metastable states. Moreover, critical
phenomena occur close to the critical boundary of metastable states. In our
case, hysteresis and metastable states are absent. Fourth, the absence of
critical correlations also suggests the SPOR transition is not a hybrid phase
transition, which combines the abrupt appearance of order, as in first order
phase transitions, with the absence of hysteresis and the presence of critical
fluctuations, as in second order phase transitions (for an example, consider
the hybrid transition in the Kuramoto model with frequency-degree correlations
[8].) Fifth, despite the absence of symmetry breaking, states above and below
the critical point of the SPOR transition are topologically distinct states
characterized by different winding numbers.

Within the concept of topological phase transition, states of matter can be
classified according to their topological properties [15, 11, 12, 13]. In
contrast to the conventional Landau paradigm of order parameters associated
with spontaneous symmetry breaking, topological phases of matter are
characterized by non-local topological invariants, such as Chern numbers for
strongly correlated topological phases in ultracold gases [13] and topological
insulators [12], or winding numbers for topological defects [15]. Topological
phase transitions occur without symmetry breaking between states with
different topological properties.

Our analysis of the SPOR transition has shown that a singular point in the
order-parameter space, corresponding to the state with zero order parameter,
plays a crucial role in the topological transition. This singular point lies
outside the limit cycle of oscillations, but is encircled by limit cycle in
the rotating phase. As a result, the limit cycles corresponding to
oscillations and rotations acquire different topological properties and form
two topologically distinct classes corresponding to different winding numbers.
The limit cycles of oscillations have winding number zero and the limit cycles
of rotations have winding number $\pm 1$ depending on the sign of the detuning
parameter. Limit cycles belonging to the same class can be smoothly
(adiabatically) deformed one into another, while cycles belonging to different
classes cannot be deformed smoothly one into another. Based on this analysis,
we conclude that the SPOR transition is a topological phase transition between
states with winding numbers 0 and $\pm 1$. Since the transition is topological
in origin, it is not caused by a specific bifurcation, and was therefore
missed by previous studies based on bifurcation theory [9, 17, 10]. A similar
phase transition has also been reported in the Kuramoto model with interacting
identical phase oscillators subjected to white noise [18].

As mentioned in the Introduction, topological phases of matter have been found
in quantum systems, namely in fractional quantum Hall liquids and topological
isolators, see, for example, reviews [11, 13]. There is an interesting
similarity between the SPOR transition in the classical forced Kuramoto model
and a transition from the flipper phase to the spinner phase in a mechanical
model of topological metamaterials [19] (compare our Fig. 2 and Fig. 4(c) in
paper [19]). Topological states with winding number $\pm 1$ and 0 on Fig. 2
are topologically similar to topological edge states in two-dimensional
topolectrical circuits [20] where bound two photon topological states appear
if the topoelectrical circuit has a unit cell characterized by winding number
1, but are absent if the winding number is zero (see Fig. 3 in paper [20]).
Note that these topological metamaterials and topolelectrical circuits are
classical analogs of quantum systems with topological edge states [19, 20].

The SPOR transition is an example of a topological transition between two
distinct topological phases with winding numbers $\pm 1$ and zero in a
classical system of interacting phase oscillators with a singular point in the
order-parameter space. We believe this kind of topological transition may be
found in other classical systems. As an illustrative example of a SPOR
transition, we may consider a motorboat making circles on the open sea. All
circles are topologically equivalent independently of their position. Let us
now introduce an observer in the water. The situation becomes crucially
different. If the observer is placed outside these circles, then the motorboat
oscillates with respect to the observer. The observer can track the motorboat
just by turning their neck. If the motorboat encircles the observer, rotating
around them, the observer must also rotate their body to track the motorboat.
The critical circle corresponds to the situation when the motorboat passes
through the observer. Assuming the person steering the motorboat wishes to
avoid committing a crime, passing through the observer is forbidden. In this
model, the observer plays the role of the singular point. States with the
observer outside and inside the closed tracks created by the motorboat have
different topological properties with respect to the observer, i.e. winding
numbers 0 and $\pm 1$, respectively. A transition from one state to another is
an abrupt topological transition.

## Declaration of Competing Interest

None.

## Credit Author Statement

All authors contributed equally to this paper.

## Acknowledgments

We thank Ricardo Guimarães Dias for useful discussions. This work is funded by
national funds (OE), through Portugal’s FCT Fundação para a Ciência e
Tecnologia, I.P., within the scope of the framework contract foreseen in
paragraphs 4, 5 and 6 of article 23, of Decree-Law 57/2016, of August 29, and
amended by Law 57/2017, of July 19. E. A. P. W. acknowledges the financial
support provided by FCT under PhD grant SFRH/BD/121331/2016.

## References

  * [1] Y. Kuramoto, International symposium on mathematical problems in theoretical physics, Lecture Notes in Physics 30 (1975) 420.
  * [2] S. H. Strogatz, From Kuramoto to Crawford: Exploring the onset of synchronization in populations of coupled oscillators, Physica D: Nonlinear Phenomena 143 (1) (2000) 1–20.  doi:10.1016/S0167-2789(00)00094-4.
  * [3] J. A. Acebrón, L. L. Bonilla, C. J. P. Vicente, F. Ritort, R. Spigler, The kuramoto model: A simple paradigm for synchronization phenomena, Reviews of modern physics 77 (1) (2005) 137.
  * [4] A. Arenas, A. Díaz-Guilera, J. Kurths, Y. Moreno, C. Zhou, Synchronization in complex networks, Physics Reports 469 (3) (2008) 93–153.  doi:10.1016/j.physrep.2008.09.002.
  * [5] F. A. Rodrigues, T. K. D. Peron, P. Ji, J. Kurths, The Kuramoto model in complex networks, Physics Reports 610 (Supplement C) (2016) 1–98.  doi:10.1016/j.physrep.2015.10.008.
  * [6] D. Abrams, S. Strogatz, Chimera states for coupled oscillators, Physical Review Letters 93 (17) (2004) 174102.
  * [7] T. Qiu, S. Boccaletti, I. Bonamassa, Y. Zou, J. Zhou, Z. Liu, S. Guan, Synchronization and bellerophon states in conformist and contrarian oscillators, Scientific Reports 6 (2016) 36713.
  * [8] B. Coutinho, A. Goltsev, S. Dorogovtsev, J. Mendes, Kuramoto model with frequency-degree correlations on complex networks, Physical Review E 87 (3) (2013) 032106.
  * [9] H. Sakaguchi, Cooperative Phenomena in Coupled Oscillator Systems under External Fields, Progress of Theoretical Physics 79 (1) (1988) 39–46.  doi:10.1143/PTP.79.39.
  * [10] L. M. Childs, S. H. Strogatz, Stability diagram for the forced Kuramoto model, Chaos: An Interdisciplinary Journal of Nonlinear Science 18 (4) (2008) 043128\.  doi:10.1063/1.3049136.
  * [11] X.-G. Wen, Topological orders and edge excitations in fractional quantum hall states, Advances in Physics 44 (5) (1995) 405–473.
  * [12] M. Z. Hasan, C. L. Kane, Colloquium: topological insulators, Reviews of modern physics 82 (4) (2010) 3045.
  * [13] N. Goldman, J. C. Budich, P. Zoller, Topological quantum matter with ultracold gases in optical lattices, Nature Physics 12 (7) (2016) 639–645.
  * [14] E. Ott, T. M. Antonsen, Low dimensional behavior of large systems of globally coupled oscillators, Chaos: An Interdisciplinary Journal of Nonlinear Science 18 (3) (2008) 037113.  doi:10.1063/1.2930766.
  * [15] N. D. Mermin, The topological theory of defects in ordered media, Reviews of Modern Physics 51 (3) (1979) 591.
  * [16] L. D. Landau, E. M. Lifshitz, Statistical Physics: Volume 5, Vol. 5, Elsevier, 2013\.
  * [17] T. M. Antonsen, R. T. Faghih, M. Girvan, E. Ott, J. Platig, External periodic driving of large systems of globally coupled phase oscillators, Chaos: An Interdisciplinary Journal of Nonlinear Science 18 (3) (2008) 037112.  doi:10.1063/1.2952447.
  * [18] M. A. Zaks, A. B. Neiman, S. Feistel, L. Schimansky-Geier, Noise-controlled oscillations and their bifurcations in coupled phase oscillators, Physical Review E 68 (6) (2003) 066206.
  * [19] B. G.-g. Chen, N. Upadhyaya, V. Vitelli, Nonlinear conduction via solitons in a topological mechanical insulator, Proceedings of the National Academy of Sciences 111 (36) (2014) 13004–13009.
  * [20] N. A. Olekhno, E. I. Kretov, A. A. Stepanenko, P. A. Ivanova, V. V. Yaroshenko, E. M. Puhtina, D. S. Filonov, B. Cappello, L. Matekovits, M. A. Gorlach, Topological edge states of interacting photon pairs emulated in a topolectrical circuit, Nature Communications 11 (1) (2020) 1–8.


Guided Flows for Generative Modeling and Decision Making

Qinqing Zheng∗ Matt Le∗ Neta Shaul† Yaron Lipman∗† Aditya Grover‡ Ricky T. Q.
Chen∗

∗FAIR, Meta †Weizmann Institute of Science $\ddagger$UCLA

###### Abstract

Classifier-free guidance is a key component for improving the performance of
conditional generative models for many downstream tasks. It drastically
improves the quality of samples produced, but has so far only been used for
diffusion models. Flow Matching (FM), an alternative simulation-free approach,
trains Continuous Normalizing Flows (CNFs) based on regressing vector fields.
It remains an open question whether classifier-free guidance can be performed
for Flow Matching models, and to what extent does it improve performance. In
this paper, we explore the usage of Guided Flows for a variety of downstream
applications involving conditional image generation, speech synthesis, and
reinforcement learning. In particular, we are the first to apply flow models
to the offline reinforcement learning setting. We also show that Guided Flows
significantly improves the sample quality in image generation and zero-shot
text-to-speech synthesis, and can make use of drastically low amounts of
computation without affecting the agent’s overall performance.

##  1 Introduction

Conditional generative modeling paves the way to numerous machine learning
applications such as conditional image generation (Dhariwal and Nichol, 2021;
Rombach et al., 2022), text-to-speech synthesis (Wang et al., 2023; Le et al.,
2023), and even solving decision making problems (Chen et al., 2021; Janner et
al., 2021, 2022; Ajay et al., 2022). Models that appear ubiquitously across a
variety of application domains are diffusion models (Sohl-Dickstein et al.,
2015; Ho et al., 2020) and flow-based models (Song et al., 2020; Lipman et
al., 2023; Albergo and Vanden-Eijnden, 2022). While flow models have been
shown to be more efficient alternatives over diffusion models (Lipman et al.,
2023; Pooladian et al., 2023) in unconditional generation, requiring less
compute costs to sample, their behavior in conditional generation tasks has
not been explored as much.

In this work, we aim to study the behavior of Flow Matching models for
conditional generation, exploring a variety of applications ranging from
generative modeling over multiple modalities, to reinforcement learning and
planning. In particular, we discuss the adoption of classifier-free guidance
(Ho and Salimans, 2022) to the case of Flow Matching models, showing that an
analogous modification can be made to the velocity vector fields, including
the optimal transport (Lipman et al., 2023) and cosine scheduling (Albergo and
Vanden-Eijnden, 2022) flows used by prior works.

We experimentally validate Guided Flows in diverse application domains, from
image generation, to zero-shot text-to-speech, to offline reinforcement
learning, see Table 1.1. We show that Guided Flows retain many of the
characteristics of regular Flow Matching models, in that they can achieve
favorable compute efficiency and performance tradeoffs, while significantly
improving the sampling quality of the model. Additionally, for offline
reinforcement learning (RL) problems, we are the first to test the use of
flows to our knowledge. In offline RL, there is an additional difficulty in
generalizing the plans to out-of-distribution states and target return values.
Here, we find that guided flows generate reliable execution plans given the
current state and a target return values.

Application | data point $x_{1}$ | conditioning variables $y$
---|---|---
Image Generation | image | class label
Text-to-Speech | spectrogram | text & utterance
Offline RL | state sequence | target return

Table 1.1: The data point $x_{1}$ and the conditioning variables $y$ on which
the conditional guidance will be applied, for the three applications settings
that we consider.

##  2 Related Work

##### Diffusion and Flow Generative Models.

Recent major developments in generative models are on building simple models
that are highly-efficient to train while providing the means of using
conditional inference for solving downstream applications (Janner et al.,
2022; Ajay et al., 2022; Le et al., 2023). Most of this development has been
focused around diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020;
Song et al., 2020), where a form of conditional guidance (Dhariwal and Nichol,
2021; Ho and Salimans, 2022) can be applied in order to place larger emphasis
on the conditional information. Whether this form of conditional guidance can
be applied and help the performance of other types of generative models is
unclear.

The ability to query pretrained generative models to answer questions about
the world is an important research topic. While some models enable the exact
computation of such queries, casted as conditional inference, they tend to be
less accurate models (Choi et al., 2020). While conditional diffusion and flow
models have also been used for tasks such as image restoration (Kawar et al.,
2022) and noisy inversion (Pokle et al., 2023), their ability to adopt to
different modalities and domains outside of images has not been explored as
much. In particular, the usage of conditional generative models for decision
making (Janner et al., 2022; Ajay et al., 2022) often requires the model to
accurately model not only relations within the training data set but to also
generalize to novel conditional signals not seen during training.

##### Conditional Generative Modeling in Offline Reinforcement Learning.

Reinforcement learning (RL) is a powerful paradigm that has been widely
applied to solve complex sequential decision making tasks, such as playing
games Silver et al. (2016), controlling robotics Kober et al. (2013), dialogue
systems Li et al. (2016); Singh et al. (1999). The premise of _offline RL_ is
to learn effective policies solely from static datasets that consist of
previous collected experiences, generated by certain unknown policies Levine
et al. (2020). There, the agent is not allowed to interact with the
environment, thus subsides the potential risk and cost of online interactions
for high-risk domains such as healthcare.

Conditional generative models are becoming handy tools to help decision
making. A rich body of work Chen et al. (2021); Janner et al. (2021, 2022);
Ajay et al. (2022); Zhao and Grover (2023); Zheng et al. (2023) focuses on
modeling trajectories as sequences of state, action, and reward tokens.
Instead of optimizing the expected return as the classic RL methods, the
training objective of all these methods is to simply maximize the likelihood
of sequences in the offline dataset. In essence, these methods cast RL as
supervised sequence modeling problems, and generative models such as
Transformer Vaswani et al. (2017); Radford et al. (2018), Diffusion Models Ho
et al. (2020) thus come into play. Solving RL from this new perspective
improves its training stability, and further opens the door of multimodal and
multitask pretraining Zheng et al. (2022); Lee et al. (2022); Reed et al.
(2022), similar to other domains like language and vision Radford et al.
(2018); Chen et al. (2020); Brown et al. (2020); Lu et al. (2022). There, a
generative model can be used as a policy to autoregressively generate actions
Chen et al. (2021); Zheng et al. (2022); alternatively, it can generate
imagined trajectories which serves as execution plans given current state and
a target output such as return or goal location Janner et al. (2021, 2022);
Ajay et al. (2022); Zhao and Grover (2023). There are other works that replace
the parameterized Gaussian policies in classic RL methods by generative models
to facilitate the modeling of multimodal action distributions Wang et al.
(2022); Chi et al. (2023).

##  3 Guided Flow Matching

##### Setup.

Let $(x_{1},y)\sim q(x_{1},y)$ denote a true data point where
$y\in\mathbb{R}^{k}$ is a conditioning variable; $x_{0}\sim p(x_{0})$ is a
noise sample, where both data and noise reside in the same euclidean space,
i.e., $x_{0},x_{1}\in\mathbb{R}^{d}$. Continuous Normalizing Flows (CNFs; Chen
et al. 2018) define a map taking the noise sample $x_{0}$ to a data sample
$x_{1}$ by first learning a vector field
$u:[0,1]\times\mathbb{R}^{d}\times\mathbb{R}^{k}\rightarrow\mathbb{R}^{d}$ and
second, integrating the vector field, i.e., solving the following Ordinary
Differential Equation (ODE)

$\dot{x}_{t}=u_{t}(x_{t}|y)$ (1)

starting at $x_{0}$ for $t=0$ and solving until time $t=1$. For notation
simplicity, we use subscript $t$ to denote the first input of function $u$.

##### Flow Matching (FM).

Lipman et al. (2023); Albergo and Vanden-Eijnden (2022) propose a method for
efficient training of $u$ based on regressing a _target velocity field_. This
target velocity field, denoted by
$u:[0,1]\times\mathbb{R}^{d}\times\mathbb{R}^{k}\rightarrow\mathbb{R}^{d}$,
takes noise samples $x_{0}$ and condition vectors $y$ to data samples $x_{1}$,
i.e., it is constructed to generate the following _marginal probability path_

$p_{t}(x|y)=\int p_{t}(x|x_{1})q(x_{1}|y)dx_{1},$ (2)

where $p_{t}(\cdot|x_{1})$ is a probability path interpolating between noise
and a single data point $x_{1}$. That is, $p_{t}(\cdot|x_{1})$ satisfies

$p_{0}(\cdot|x_{1})\equiv p(\cdot),\qquad
p_{1}(\cdot|x_{1})\approx\delta_{x_{1}}(\cdot),$ (3)

where $\delta_{x_{1}}(\cdot)$ is the delta probability that concentrates all
its mass at $x_{1}$. An immediate consequence of the boundary conditions in
equation (3) is that $p_{t}(\cdot|y)$ indeed interpolates between noise and
data, i.e., $p_{0}(\cdot|y)\equiv p(\cdot)$ and $p_{1}(\cdot|y)\approx
q(\cdot|y)$ for all $y$.

Next, we assume $u_{t}(\cdot|x_{1})$ is a velocity field that _generates_
$p_{t}(\cdot|x_{1})$ in the sense that solutions to equation (1), with
$u_{t}(\cdot|x_{1})$ as the velocity field and $x_{0}\sim p(x_{0})$, satisfy
$x_{t}\sim p_{t}(x_{t}|x_{1})$. The target velocity field for FM is then
defined via

$u_{t}(x|y)=\int
u_{t}(x|x_{1})\frac{p_{t}(x|x_{1})q(x_{1}|y)}{p_{t}(x)}dx_{1},$ (4)

and can be proved to generate, in the sense described above, the _marginal
probability path_ $p_{t}(\cdot|y)$ in equation (2). FM is trained by
minimizing a tractable loss called the Conditional Flow Matching (CFM) loss
(defined later), whose global minimizer is the target velocity field $u_{t}$.

##### Gaussian Paths.

A popular instantiation of paths $p_{t}(x|x_{1})$ are _Gaussian paths_ defined
by

$p_{t}(x|x_{1})={\mathcal{N}}(x|\alpha_{t}x_{1},\sigma_{t}^{2}I),$ (5)

where ${\mathcal{N}}$ is the Gaussian kernel,
$\alpha,\sigma:[0,1]\rightarrow[0,1]$ are differentiable functions satisfying
$\alpha_{0}=0=\sigma_{1}$, $\alpha_{1}=1=\sigma_{0}$111As before, we use
subscript $t$ to denote the input of $\alpha$ and $\sigma$.. A pair
$(\alpha_{t},\sigma_{t})$ is called a _scheduler_. Marginal paths $p_{t}(x|y)$
defined with $p_{t}(x|x_{1})$ as in equation (5) are called _marginal Gaussian
paths_.

By convention we denote by $\varnothing$ the null conditioning and set
$q(x):=q(x|\varnothing)$, $u^{\theta}_{t}(x):=u^{\theta}_{t}(x|\varnothing)$,
and $u_{t}(x):=u_{t}(x|\varnothing)$. Second

##### Guided Flows.

Next, we adapt the notion of Classifier-Free Guidance (CFG; Ho and Salimans
2022) to conditional velocity fields $u_{t}(x|y)$. As in CFG, we set our goal
to sample from the distribution $\tilde{q}(x|y)\propto
q(x)^{1-\omega}q(x|y)^{\omega}$, $\omega\in\mathbb{R}$, where only explicit
Flow Matching models for $q(x)$ and $q(x|y)$ are given. Motivated by CFG we
define

$\tilde{u}_{t}(x|y)=(1-\omega)u_{t}(x)+\omega u_{t}(x|y)$ (6)

To justify this formula for velocity fields, we first relate $u_{t}(x|y)$ to
the score function $\nabla\log p_{t}(x|y)$ using the following lemma proved in
Appendix A.1:

###### Lemma 1

Let $p_{t}(x|y)$ be a Gaussian Path defined by a scheduler
$(\alpha_{t},\sigma_{t})$, then its generating velocity field $u_{t}(x|y)$ is
related to the score function $\nabla\log p_{t}(x|y)$ by

$u_{t}(x|y)=a_{t}x+b_{t}\nabla\log p_{t}(x|y),$ (7)

where

$a_{t}=\frac{\dot{\alpha}_{t}}{\alpha_{t}},\qquad
b_{t}=(\dot{\alpha}_{t}\sigma_{t}-\alpha_{t}\dot{\sigma}_{t})\frac{\sigma_{t}}{\alpha_{t}}.$
(8)

Next, using this the lemma and plugging equation (7) for
$u_{t}(x)=u_{t}(x|\varnothing)$ and $u_{t}(x|y)$ in the r.h.s. of equation
(6), we get

$\displaystyle\tilde{u}_{t}(x|y)=a_{t}x+b_{t}\nabla\log\tilde{p}_{t}(x|y),$
(9)

where

$\tilde{p}_{t}(x|y)\propto p_{t}(x)^{1-\omega}p_{t}(x|y)^{\omega}$ (10)

is the geometric weighted average of $p_{t}(x)$ and $p_{t}(x|y)$.

As we prove in Appendix B, this velocity field $\tilde{u}_{t}$ coincides with
the one in the Probability Flow ODE (Song et al., 2020) used in Classifier
Free Guidance for approximate sampling from distribution $\tilde{q}(\cdot|y)$.
This provides a justification for equation (6). We note, however, that this
analysis shows that both Guided Flows and CFG are guaranteed to sample from
$\tilde{q}(\cdot|y)$ at time $t=1$ if the probability path
$\tilde{p}_{t}(\cdot|y)$ defined in equation (10) is close to the marginal
probability path $\int p_{t}(\cdot|x_{1})\tilde{q}(x_{1}|y)dx_{1}$, but it is
not clear to what extent this assumption holds in practice.

Input: $p_{\text{uncond}}$ probability of unconditional training

Initialize $u_{t}^{\theta}$

while  _not converged_ do

$(x_{1},y)\sim q(x_{1},y)$ $\triangleright$ sample data and condition

$y\leftarrow\varnothing$ with probability $p_{\text{uncond}}$ $\triangleright$
null condition

$x_{0}\sim p(x_{0})$ $\triangleright$ sample noise

$x_{t}\leftarrow\alpha_{t}x_{1}+\sigma_{t}x_{0}$ $\triangleright$ noisy data
point

$\dot{x}_{t}\leftarrow\dot{\alpha}_{t}x_{1}+\dot{\sigma}_{t}x_{0}$
$\triangleright$ derivative of noisy data point

Take gradient step on
$\nabla_{\theta}\left\|u^{\theta}_{t}(x_{t},y)-\dot{x}_{t}\right\|^{2}$

Output: $u_{t}^{\theta}$

Algorithm 1 Training Guided Flows

##### Training Guided Flows.

Training guided flow follows the practice in CFG but replaces the Diffusion
training loss with the Conditional Flow Matching (CFM) loss (Lipman et al.,
2023). This leads to following loss function:

${\mathcal{L}}(\theta)=\mathbb{E}_{t,b,q(x_{1},y),p(x_{0})}\left\|u^{\theta}_{t}(x_{t}|(1-b)\cdot
y+b\cdot\varnothing)-\dot{x}_{t}\right\|^{2}$,

(11)

where $t$ is sampled uniformly in $[0,1]$,
$b\sim\text{Bernoulli}(p_{\text{uncond}})$ is used to indicate whether we will
use null condition, $x_{1}$ and $y$ are sampled from the true data
distribution, $x_{t}=\alpha_{t}x_{1}+\sigma_{t}x_{0}$,
$\dot{x}_{t}=u_{t}(x_{t}|x_{1})=\dot{\alpha}_{t}x_{1}+\dot{\sigma}_{t}x_{0}$
and
$u^{\theta}:[0,1]\times\mathbb{R}^{d}\times\mathbb{R}^{k}\rightarrow\mathbb{R}^{d}$
is a neural network with learnable parameters $\theta\in\mathbb{R}^{p}$. The
training process is summarized in Algorithm 1.

##### Sampling Guided Flows.

Sampling from Guided Flows required approximating the solution to the sampling
ODE (equation (1)) with the guided velocty field $\tilde{u}_{t}$ defined in
equation (6), see Algorithm 2.

##### Illustrative Example.

Figure 3.1 shows a visualization of the effect of Guided Flows in a toy 2D
example of a mixture of Gaussian distributions, where $y$ is the latent
variable specifying the identity of the mixture component, and $q(x_{1}|y)$ is
one single Gaussian component. When the guidance weight is $1.0$, there is no
guidance performed, and we are sampling from the unconditional marginal
$q(x_{1})$. We see that as guidance weight increases, the samples move away
from the unconditional distribution $q(x_{1})$.

Input: trained velocity field $u^{\theta}_{t}$, condition $y$, guidance
parameter $\omega$, number of ODE steps $n_{\text{ode}}$

$x_{0}\sim p(x_{0})$ $\triangleright$ sample noise

$h\leftarrow\frac{1}{n_{\text{ode}}}$ $\triangleright$ step size

$\tilde{u}_{t}(\cdot)\leftarrow(1-\omega)u^{\theta}_{t}(\cdot)+\omega
u^{\theta}_{t}(\cdot|y)$ $\triangleright$ guided velocity

for  _$t=0,h,\ldots,1-h$_ do

$x_{t+h}\leftarrow$ ODEStep($\tilde{u}_{t}$, $x_{t}$)  $\triangleright$ ODE
solver step

Output: $x_{1}$

Algorithm 2 Sampling from Guided Flows

$w=1.0$

$w=2.0$

$w=3.0$

$w=4.0$

Figure 3.1: The effect of increasing guidance weight.

##  4 Conditional Generative Modeling

In this section, we perform experiments to test whether Guided Flows can help
achieve better sample quality than unguided models. In particular, we consider
two application settings: conditional image generation and zero-shot text-to-
speech synthesis. The goal of these experiments to test Guided Flows on
standard generative modeling settings, where the evaluation is purely based on
sample quality, while exploring different data modalities, before we move on
validating Guided Flows for more complex planning tasks in Section 5.

###  4.1 Conditional Image Generation

1.0

Brambling

Husky

2.0

3.0

1.0

Meerkat

Otter

2.0

3.0

Figure 4.1: Generated samples from face-blurred ImageNet using four different
class labels, showing the effect of different guidance weights with Guided
Flows. Higher guidance weights result in more class-specific features to
appear.

We use the official _face-blurred_ ImageNet data and then downsample to 64×64
using the open source preprocessing scripts from Chrabaszcz et al. (2017). We
then train Guided Flow Matching models $p_{t}(x|y)$ where $x$ denotes an image
and $y$ is the class label of that image, using the same architecture and
hyperparameter settings as in (Dhariwal and Nichol, 2021). As a baseline, we
consider a diffusion model (DDPM; Ho and Salimans 2022) guided using
classifier-free guidance (Ho and Salimans, 2022). For Guided Flows, we
consider two affine Gaussian probability paths, the optimal transport (FM-OT)
path considered by Lipman et al. (2023) and the cosine scheduling (FM-CS) path
considered by Albergo and Vanden-Eijnden (2022). We note that both FM-OT and
FM-CS satisfy the assumptions of equation (5). All models were trained with
the same hyperparameters and number of iterations.

Results are displayed in Figure 4.2. We see that guidance for Flow Matching
models can drastically help increase sample quality (reducing FID from $2.54$
to $1.68$) using a midpoint solver with 200 number of function evaluations
(NFE). We do see that the optimal guidance weight can change depending on the
compute cost (NFE), with NFE=10 having a higher optimal guidance weight.
Additionally, in Figure 2(b) we show the Pareto front of the sample quality
and efficiency tradeoff, plotted for each model. We note that the optimal
guidance weight can be very different between each model, with DDPM noticeably
requiring a larger guidance weight. Here we find that FM-OT models are
slightly more efficient than the other models that we consider.

(a) Guidance

(b) Efficiency

Figure 4.2: (a) Guided Flows (FM-OT) with $w>1.0$ achieve a lower FID,
achieving $1.68$ on ImageNet-64. (b) Guided Flows with conditional optimal
transport paths seem to have the best tradeoff for efficiency.

###  4.2 Zero-shot Text-to-Speech Synthesis

Guidance Weight | Continuation | Text-only
---|---|---
Le et al. (2023) | 2.00 | 3.10
1.0 | 2.06 | 3.09
1.2 | 1.95 | 2.98
1.4 | 1.96 | 2.89
1.6 | 1.95 | 2.83
1.8 | 1.94 | 2.87
2.0 | 1.98 | 2.82
2.2 | 1.96 | 2.80
2.4 | 2.10 | 2.83
2.6 | 2.40 | 2.76
2.8 | 3.40 | 2.79
3.0 | 5.10 | 2.75
Table 4.1: Word error rates (WER) for zero-shot text-to-speech. When the
weight is $1.0$, there is no guidance performed and we sample directly from
the trained conditional distribution. We see that performance improves when
using Guided Flows with weights greater than 1.0.

Given a target text and a transcribed reference audio as conditioning
information $y$, zero-shot text-to-speech (TTS) aims to synthesize speech
resembling the audio style of the reference, which was never seen during
training. As our Guided Flow model, we train a model on 60K hours ASR-
transcribed English audiobooks, following the experiment setup in Le et al.
(2023).

Specifically, we consider two main tasks. The first one is zero-shot TTS where
the first 3 seconds of each utterance is provided and the model is requested
to continue the speech. The second one is diverse speech generation, where
only the text is provided to the model. In order to assess the accuracy of the
generated results, we report the word error rate (WER) using automatic speech
recognition (ASR) models following prior works (Wang et al., 2018).

Results of Guided Flows is provided in Table 4.1, where we also provide the
results of Le et al. (2023) as reference. We also report our results when no
guidance is used (weight equal to 1.0). With guidance, we see a marginal
improvement for the continuation TTS task, and a much more sizable gain in the
text-only TTS task. This likely due to the text-only TTS task being a much
more diverse distribution.

##  5 Planning for Offline RL

###  5.1 Preliminaries

We model our environment as a Markov decision process (MDP) (Bellman, 1957)
denoted by $\langle\mathcal{S},\mathcal{A},p,P,R,\gamma\rangle$, where
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$p(s_{0})$ is the distribution of the initial state, $P(s_{t+1}|s_{t},a_{t})$
is the transition probability distribution, $R(s_{t},a_{t})$ is the
deterministic reward function, and $\gamma$ is the discount factor. At
timestep $t$, the agent observes a state $s_{t}\in\mathcal{S}$ and executes an
action $a_{t}\in\mathcal{A}$. The environment will provide the agent with a
reward $r_{t}=R(s_{t},a_{t})$, and also moves it to the next state
$s_{t+1}\sim P(\cdot|s_{t},a_{t})$. Let $\tau$ be a trajectory. For any
length-$H$ subsequence $\tau_{\text{sub}}$ of $\tau$, e.g., from timestep $t$
to $t+H-1$, we define the return-to-go (RTG) of $\tau_{\text{sub}}$ to be the
sum of its discounted return
$g(\tau_{\text{sub}})=\sum_{t^{\prime}=t}^{t+H-1}\gamma^{t^{\prime}-t}r_{t^{\prime}}$222This
is slightly different from the standard RTG definition where the discounting
factor is $\gamma^{t^{\prime}-t}$ rather than $\gamma^{t}$.. We also use
$\bm{s}(\tau_{\text{sub}})$ to denote the state sequence extracted from the
subsequence $\tau_{\text{sub}}$. A deterministic inverse dynamics model (IDM)
is a function $f:\mathcal{S}\times\mathcal{S}\mapsto\mathcal{A}$ which
predicts action using states: $\widehat{a}_{t}=f(s_{t},s_{t+1})$.

###  5.2 Our Setup

Input: trained conditional sequence model
$p_{\theta}(s_{t+1},\ldots,s_{t+H-1}|s_{t},g_{t})$, IDM
$f_{\rho}(s,s^{\prime})$, initial state $s_{0}$, initial conditioning
parameter $g_{0}$, conditioning parameter updating rule $G$

$t\leftarrow 0$

while  _episode not done_ do

Sample $(\widehat{s}_{t+1},\ldots,\widehat{s}_{t+H-1})\sim
p_{\theta}(\cdot|s_{t},g_{t})$333As reflected in Figure 5.1 and 5.2, we model
the sequence starting from $s_{t}$ rather than $s_{t+1}$ in the actual
implementation. Due to the presence of $s_{t}$, the sampling process is
slightly adjusted from Algorithm 2, as we need to zero out the vector fields
corresponding to $s_{t}$. See Algorithm 4.

Predict action $\widehat{a}_{t}=f_{\rho}(s_{t},\widehat{s}_{t+1})$

Execute $\widehat{a}_{t}$ and observe $s_{t+1}$ and $r_{t}$

$g_{t+1}\leftarrow$ compute the next conditioning parameter according to $G$

$t\leftarrow t+1$

Algorithm 3 A General Framework of Conditioned Generative Planning

Figure 5.1: The training phase of return-conditioned generative planning. The
state sequence model is trained in a classifer-free manner—with certain
probability, it is conditioned on the RTG. Figure 5.2: The evaluation phase
using return-conditioned generative planning.

We consider the paradigm proposed by Ajay et al. (2022), where a generative
model learns to predict a sequence of future states, conditioning on the
current state and target output such as expected return. Intuitively, the
predicted future states form a _plan_ to reach the target output, and the
predicted next state can be interpreted as the _next intermediate goal_ on the
roadmap. Based on the current state and the predicted next state, we use an
inverse dynamics model to predict the action to execute. Algorithm 3
summarizes this framework. We emphasize that the _target output $g_{t}$ is the
conditioning variable where guidance will perform_, and the current state
$s_{t}$ is a general conditioning variable. In this work, we consider the
target return as our conditioning variable. Return-conditioned RL methods are
widly used to solve standard RL problems where the environment produces dense
rewards Srivastava et al. (2019); Kumar et al. (2019); Schmidhuber (2019);
Emmons et al. (2021); Chen et al. (2021); Nguyen et al. (2022). Figure 5.1 and
5.2 plot the training and evaluation phases of our paradigm. Careful readers
might notice that we resample the whole sequence
$\widehat{s}_{t+1},\ldots,\widehat{s}_{t+H-1}$ at every timestep $t$, but only
use $\widehat{s}_{t+1}$ to predict $\widehat{a}_{t}$. In fact, it is
completely feasible to predict the actions for the next multiple steps using a
single plan, which also saves computation. We note that replanning at every
timestep is for the sake of planning accuracy, as the error will accumulate as
the horizon expands, and there is a tradeoff between computational efficiency
and agent performance. For diffusion models, previous works have used
heuristics to improve the execution speed, e.g., reusing previously generated
plans to warm-start the sampling of subsequence plans Janner et al. (2022);
Ajay et al. (2022). This is beyond the scope of our paper, and we only
consider replanning at every timestep for simplicity.

###  5.3 Experiments

Our experiments aim at answering the following questions:

  1. Q1.

Can conditional flows generate meaningful plans for RL problems given a target
return?

  2. Q2.

How do flows compare to diffusion models, in terms of both downstream RL task
performance and compute efficiency?

  3. Q3.

Compared with unguided flows, can guidance help planning?

##### Tasks and Datasets

We consider three Gym locomotions tasks, hopper, walker and halfcheetah, using
offline datasets from the D4RL benchmark Fu et al. (2020). For all the
experiments, we train 5 instances of each method with different seeds. For
each instance, we run 20 evaluation episodes. We shall discuss the model
architectures and important hyperparameters below, and we refer the readers to
Appendix D for more details.

##### Baseline

We compare our method to Decision Diffuser Ajay et al. (2022), which uses the
same paradigm with diffusion models to model the state sequences.

##### Sequence Model Architectures

For the locomotion tasks, we train flows for state sequences of length $H=64$,
parameterized through the velocity field $u^{\theta}$. Similar to the previous
work Janner et al. (2022); Ajay et al. (2022), the velocity field $u^{\theta}$
is modeled as a temporal U-Net consisting of repeated convolutional residual
blocks. Both the time $t$ and the RTG $g(\bm{s})$ are projected to latent
spaces via multilayer perceptrons, where $t$ is first transformed to its
sinusoidal position encoding. For the baseline method, the diffusion model is
training similarly with $200$ diffusion steps, where we use a temporal U-Net
to predict the noise at each diffusion steps throughout the diffusion process.

##### Inverse Dynamic Model

For all the environments and datasets, we model the IDM by an MLP with 2
hidden layers and 1024 hidden units per layer. Among all the offline
trajectories, we randomly sample 10% of them as the validation set. We train
the IDM for 100k iterations, and use the one that yields the best validation
performance.

##### Low Temperature Sampling

To sample from the diffusion model, we use $200$ diffusion steps. For the sake
of fair comparison, we also use $200$ ODE steps when sampling from the flow
matching model. Following Ajay et al. (2022), we use the _low temperature
sampling_ technique for diffusion model, where at each diffusion step $k$ we
sample the state sequence from
$\mathcal{N}(\widehat{\mu}_{k},\alpha^{2}\widehat{\Sigma}_{k})$444$\widehat{\mu}_{k}$
and $\widehat{\Sigma}_{k}$ are the predicted mean and variance for sampling at
the $k$th diffusion step. We refer the readers to Ho et al. (2020) for more
details. with a hand-selected temperature parameter $\alpha\in(0,1)$. We sweep
over 3 values of $\alpha$ for all our experiments: $0.1,0.25$, and $0.5$. For
flow matching, we analogously set the initial distribution
$p_{0}(x_{0})=\mathcal{N}(0,\nu^{2}I)$ and we sweep over two values of $\nu$:
$0.1$ and $1$.

| hopper | walker | halfcheetah | Average
---|---|---|---|---
| medium-replay | medium | medium-expert | medium-replay | medium | medium-expert | medium-replay | medium | medium-expert
Flow | 0.89 | 0.84 | 1.05 | 0.78 | 0.77 | 0.94 | 0.42 | 0.49 | 0.97 | 0.79
Diffusion | 0.87 | 0.72 | 1.09 | 0.64 | 0.8 | 1.07 | 0.48 | 0.41 | 0.95 | 0.78

Table 5.1: The normalized return obtained when the state sequence models are
trained by flows and diffusion models. We use the same IDM for both methods.
Results aggregated over 5 training instances with different seeds.

####  5.3.1 Plan Generation (Q1)

To verify the capability of flows to generate meaningful plans that can guide
the agent, we sample from a flow on the hopper-medium dataset. We randomly
select a subtrajectory $\tau_{\text{sub}}$ from the dataset, and let flow
condition on its first state and RTG $0.8$. This RTG value is out-of-
distribution, as the maximum RTG value of the training dataset is $0.61$.
Figure 5.3 plots both state sequences. We can see that the generated state
sequence is almost identical to the ground truth, even though the RTG value we
condition on is out-of-distribution. See Figure E.2 for more examples.

Figure 5.3: (Top) A truth state sequence randomly sampled from the hopper-
medium dataset. (Bottom) Flow generated state sequence, conditioning on the
first state and a large out-of-distribution RTG. The guidance weight is $3.0$.
Figure 5.4: The obtained normalized return when the internal samplling steps
used to generate the state sequence varies.

####  5.3.2 Benchmark (Q2)

In addition to the temperature parameter for sampling, for both methods, we
sweep over 3 values of RTG: 0.4, 0.5, and 0.6 for the medium-replay and medium
datasets of halfcheetah, and 0.7, 0.8 and 0.9 for all the other datasets. We
also sweep over 4 values of guidance parameters: $1.8,2.0,2.2,2.4$ for
diffusion models, and $1.0,1.5,2.0,2.5$ for flows. We train both guided flows
and guided diffusion models for 2 million iterations, and save checkpoints
every $200k$ iteration. We evaluate the performance on all the saved
checkpoints and report the best result in Table 5.1555This is because Ajay et
al. (2022) report the best results over a collection of checkpoints.. The
performances of flow matching and diffusion model are comparable, where flow
matching performs marginally better.

##### Computational Efficiency

A well-known pain-point of diffusion model is its computational inefficiency.
In our case, it takes $200$ internal sampling steps to generate a sample with
full quality. Many approximation methods have been proposed to speed up the
sampling process, including distillation Salimans and Ho (2022), noise
scheduling Nichol and Dhariwal (2021); Lin et al. (2023), etc. As a tradeoff,
these methods all lead to quality loss. To understand the computation-vs-
quality tradeoff, we run an ablation experiment for the hopper task, where we
only sample with $K$ internal steps $(K\leq 200)$. Figure 5.4 plots the
normalized return we obtain versus the number of internal steps. For all 3
datasets, phase transitions occur for both methods. Surprisingly, $10$ ODE
steps are sufficient for flows to generate samples leading to the same return
as $200$ ODE steps; whereas diffusion models need $150$ diffusion steps.
Figure 5.5 shows that the CPU time consumed by the diffusion model and the
flow are roughly the same when the number of internal steps match. Moreover,
the CPU time scales linearly as the number of internal steps increase. This
means, compared with diffusion model, flows only need $10/150\approx 6.67\%$
computing time to generate samples leading to the same downstream performance.
The trend remains the same when the batch size increase, see Figure E.1.

Figure 5.5: The CPU time of generating a batch of $704$-dimensional vectors
(the subsequence length is 64, and each state is of dimension 11), where the
batch size is $20$. Results averaged over $100$ replications.

####  5.3.3 Influence of Guidance (Q3)

Table 5.2 reports the normalized return obtained by our agents for the hopper
and halfcheetah tasks, where the guidance weight varies from 1.0 to 3.0. The
flows are trained on both medium and medium-replay datasets for hopper, and
the medium dataset for halfcheetah. The guidance weight $1.0$ yields unguided
flows. The results show that the guided flows outperform the unguided ones on
all three datasets.

Guidance Weight | hopper | halfcheetah
---|---|---
medium | medium-replay | medium
1.0 | 0.64 | 0.83 | 0.48
1.5 | 0.73 | 0.87 | 0.48
2.0 | 0.81 | 0.89 | 0.49
2.5 | 0.81 | 0.86 | 0.46
3.0 | 0.84 | 0.84 | 0.45

Table 5.2: The normalized return obtained when using different guidance
weights. For both tasks, guided flows outperform unguided flows (guidance
weight is $1$).

##  6 Conclusion and Discussion

We empirically validate the conditional generative capabilities of flow-based
models trained through recently-proposed simulation-free algorithms (Lipman et
al., 2023; Albergo and Vanden-Eijnden, 2022). In doing so, we show that
conditional guidance can lead to better results for flow-based models, and
that flow models perform just as well as diffusion in generalizing to rare or
unseen conditional signals, as is the case for reinforcement learning tasks.
This is all while keeping the sampling efficiency inherent in flow models,
allowing us to significantly reduce sampling costs while performing just as
well on many downstream applications, particularly in plan generation.

#### Acknowledgments

The authors thank Zihan Ding, Maryam Fazel-Zarandi, Brian Karrer, Maximilian
Nickel, Mike Rabbat, Yuandong Tian, Amy Zhang, and Siyan Zhao for insightful
discussions.

## References

  * Ajay et al. (2022) Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal.  Is conditional generative modeling all you need for decision-making?  _arXiv preprint arXiv:2211.15657_ , 2022.
  * Albergo and Vanden-Eijnden (2022) Michael S Albergo and Eric Vanden-Eijnden.  Building normalizing flows with stochastic interpolants.  _arXiv preprint arXiv:2209.15571_ , 2022.
  * Bellman (1957) Richard Bellman.  A markovian decision process.  _Indiana Univ. Math. J._ , 1957.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  In _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=a7APmM4B9d.
  * Chen et al. (2020) Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _International conference on machine learning_ , pages 1691–1703. PMLR, 2020.
  * Chen et al. (2018) Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  _Advances in neural information processing systems_ , 31, 2018.
  * Chi et al. (2023) Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song.  Diffusion policy: Visuomotor policy learning via action diffusion.  _arXiv preprint arXiv:2303.04137_ , 2023.
  * Choi et al. (2020) Y Choi, Antonio Vergari, and Guy Van den Broeck.  Probabilistic circuits: A unifying framework for tractable probabilistic models.  _UCLA. URL: http://starai. cs. ucla. edu/papers/ProbCirc20. pdf_ , 2020.
  * Chrabaszcz et al. (2017) Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter.  A downsampled variant of imagenet as an alternative to the cifar datasets.  _arXiv preprint arXiv:1707.08819_ , 2017.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Emmons et al. (2021) Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine.  Rvs: What is essential for offline rl via supervised learning?  _arXiv preprint arXiv:2112.10751_ , 2021.
  * Fu et al. (2020) Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Ho and Salimans (2022) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Janner et al. (2021) Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  In _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=wgeK563QgSw.
  * Janner et al. (2022) Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine.  Planning with diffusion for flexible behavior synthesis.  _arXiv preprint arXiv:2205.09991_ , 2022.
  * Kawar et al. (2022) Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  _Advances in Neural Information Processing Systems_ , 35:23593–23606, 2022.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma et al. (2023) Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models, 2023.
  * Kober et al. (2013) Jens Kober, J Andrew Bagnell, and Jan Peters.  Reinforcement learning in robotics: A survey.  _The International Journal of Robotics Research_ , 32(11):1238–1274, 2013.
  * Kumar et al. (2019) Aviral Kumar, Xue Bin Peng, and Sergey Levine.  Reward-conditioned policies.  _arXiv preprint arXiv:1912.13465_ , 2019.
  * Le et al. (2023) Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al.  Voicebox: Text-guided multilingual universal speech generation at scale.  _Advances in neural information processing systems_ , 2023.
  * Lee et al. (2022) Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al.  Multi-game decision transformers.  _Advances in Neural Information Processing Systems_ , 35:27921–27936, 2022.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky.  Deep reinforcement learning for dialogue generation.  _arXiv preprint arXiv:1606.01541_ , 2016.
  * Lin et al. (2023) Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang.  Common diffusion noise schedules and sample steps are flawed.  _arXiv preprint arXiv:2305.08891_ , 2023.
  * Lipman et al. (2023) Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.  Flow matching for generative modeling.  _International Conference on Learning Representations_ , 2023.
  * Lu et al. (2022) Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch.  Pretrained transformers as universal computation engines.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * Mish (2019) Misra D Mish.  A self regularized non-monotonic activation function [j].  _arXiv preprint arXiv:1908.08681_ , 2019.
  * Nguyen et al. (2022) T Nguyen, Q Zheng, and A Grover.  Reliable conditioning of behavioral cloning for offline reinforcement learning.  _arXiv preprint arXiv:2210.05158_ , 2022.
  * Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Pokle et al. (2023) Ashwini Pokle, Matthew J Muckley, Ricky TQ Chen, and Brian Karrer.  Training-free linear image inversion via flows.  _arXiv preprint arXiv:2310.04432_ , 2023.
  * Pooladian et al. (2023) Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky Chen.  Multisample flow matching: Straightening flows with minibatch couplings.  _arXiv preprint arXiv:2304.14772_ , 2023.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * Reed et al. (2022) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.  A generalist agent.  _arXiv preprint arXiv:2205.06175_ , 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Salimans and Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Schmidhuber (2019) Juergen Schmidhuber.  Reinforcement learning upside down: Don’t predict rewards–just map them to actions.  _arXiv preprint arXiv:1912.02875_ , 2019.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Singh et al. (1999) Satinder Singh, Michael Kearns, Diane Litman, and Marilyn Walker.  Reinforcement learning for spoken dialogue systems.  _Advances in neural information processing systems_ , 12, 1999.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Srivastava et al. (2019) Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber.  Training agents using upside-down reinforcement learning.  _arXiv preprint arXiv:1912.02877_ , 2019.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Wang et al. (2023) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_ , 2023.
  * Wang et al. (2018) Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  In _International conference on machine learning_ , pages 5180–5189. PMLR, 2018.
  * Wang et al. (2022) Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou.  Diffusion policies as an expressive policy class for offline reinforcement learning.  _arXiv preprint arXiv:2208.06193_ , 2022.
  * Wu and He (2018) Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Zhao and Grover (2023) Siyan Zhao and Aditya Grover.  Decision stacks: Flexible reinforcement learning via modular generative models.  _arXiv preprint arXiv:2306.06253_ , 2023.
  * Zheng et al. (2022) Qinqing Zheng, Amy Zhang, and Aditya Grover.  Online decision transformer.  In _international conference on machine learning_ , pages 27042–27059. PMLR, 2022.
  * Zheng et al. (2023) Qinqing Zheng, Mikael Henaff, Brandon Amos, and Aditya Grover.  Semi-supervised offline reinforcement learning with action-free trajectories.  In _International conference on machine learning_ , pages 42339–42362. PMLR, 2023.

##  Appendix A Proofs

###  A.1 Proof of Lemma 1

See 1

Proof (Lemma 1). The Gaussian probability path $p_{t}(x|y)$ as in equation (2)
is

$p_{t}(x|y)=\int p_{t}(x|x_{1})q(x_{1}|y)dx_{1},$ (12)

where $p_{t}(x|x_{1})={\mathcal{N}}(x|\alpha_{t}x_{1},\sigma_{t}^{2}I)$. We
express the score function as

$\displaystyle\nabla\log p_{t}(x|y)$ $\displaystyle=\frac{\nabla
p_{t}(x|y)}{p_{t}(x|y)}$ (13) $\displaystyle=\int\frac{\nabla
p_{t}(x|x_{1})q(x_{1}|y)}{p_{t}(x|y)}dx_{1}$ (14)
$\displaystyle=\int\nabla\log
p_{t}(x|x_{1})\frac{p_{t}(x|x_{1})q(x_{1}|y)}{p_{t}(x|y)}dx_{1}.$ (15)

The generating velocity field $u_{t}$ as in equation (4) is

$u_{t}(x|y)=\int
u_{t}(x|x_{1})\frac{p_{t}(x|x_{1})q(x_{1}|y)}{p_{t}(x|y)}dx_{1},$ (16)

where
$u_{t}(x|x_{1})=\frac{\dot{\sigma}_{t}}{\sigma_{t}}(x-\alpha_{t}x_{1})+\dot{\alpha}_{t}x_{1}$.
Hence, by linearity of integrals it is enough to show that

$u_{t}(x|x_{1})=\frac{\dot{\alpha}_{t}}{\alpha_{t}}x+(\dot{\alpha}_{t}\sigma_{t}-\alpha_{t}\dot{\sigma}_{t})\frac{\sigma_{t}}{\alpha_{t}}\nabla\log
p_{t}(x|x_{1}).$ (17)

And indeed,

$\displaystyle u_{t}(x|x_{1})$
$\displaystyle=\frac{\dot{\sigma}_{t}}{\sigma_{t}}(x-\alpha_{t}x_{1})+\dot{\alpha}_{t}x_{1}$
(18)
$\displaystyle=\frac{\dot{\alpha}_{t}}{\alpha_{t}}x-\frac{\dot{\alpha}_{t}}{\alpha_{t}}x+\frac{\dot{\sigma}_{t}}{\sigma_{t}}(x-\alpha_{t}x_{1})+\dot{\alpha}_{t}x_{1}$
(19)
$\displaystyle=\frac{\dot{\alpha}_{t}}{\alpha_{t}}x-(\dot{\alpha}_{t}\sigma_{t}-\alpha_{t}\dot{\sigma}_{t})\frac{1}{\alpha_{t}\sigma_{t}}\left(x-\alpha_{t}x_{1}\right)$
(20)
$\displaystyle=\frac{\dot{\alpha}_{t}}{\alpha_{t}}x+(\dot{\alpha}_{t}\sigma_{t}-\alpha_{t}\dot{\sigma}_{t})\frac{\sigma_{t}}{\alpha_{t}}\nabla\log
p_{t}(x|x_{1}),$ (21)

where in the last equality we used our assumption of Gaussian probability path
that gives $\nabla\log
p_{t}(x|x_{1})=-\frac{1}{\sigma_{t}^{2}}(x-\alpha_{t}x_{1})$. $\square$

##  Appendix B Probability Flow ODE for Scheduler $(\alpha_{t},\sigma_{t})$

In this section we provide the velocity field used in CFG (Ho and Salimans,
2022) for approximate sampling from the conditional distribution
$\tilde{q}(x|y)\propto q(x)^{1-\omega}q(x|y)^{\omega}$, $\omega\in\mathbb{R}$,
and show that it coincides with our velocity field in equation (9).

We assume the marginal probability paths (see equation (2)) $p_{t}(x)$ and
$p_{t}(x|y)$ are defined with a scheduler $(\alpha_{t},\sigma_{t})$ and data
distribution $q(x)$ and $q(x|y)$, respectively. CFG consider the probability
path

$\tilde{p}_{t}(x|y)=p_{t}(x)^{1-\omega}p_{t}(x|y)^{\omega}$ (22)

with the corresponding score function

$\nabla\log\tilde{p}_{t}(x|y)=(1-\omega)\nabla p_{t}(x)+\omega\nabla
p_{t}(x|y).$ (23)

Then the sampling is done with the Probability Flow ODE of diffusion models
(Song et al., 2020),

$\dot{x}_{t}=f_{t}x_{t}-\frac{1}{2}g_{t}^{2}\nabla\log\tilde{p}_{t}(x_{t}|y),$
(24)

where $f_{t}=\frac{d\log\alpha_{t}}{dt}$,
$g_{t}^{2}=\frac{d\sigma_{t}^{2}}{dt}-2\frac{d\log\alpha_{t}}{dt}\sigma_{t}$
(Kingma et al., 2023; Salimans and Ho, 2022). Lastly,

$\frac{d\log\alpha_{t}}{dt}=\frac{\dot{\alpha}_{t}}{\alpha_{t}}=a_{t},\quad-\frac{1}{2}\frac{d\sigma_{t}^{2}}{dt}+\frac{d\log\alpha_{t}}{dt}\sigma_{t}=(\dot{\alpha}_{t}\sigma_{t}-\alpha_{t}\dot{\sigma}_{t})\frac{\sigma_{t}}{\alpha_{t}}=b_{t},$
(25)

plugging this in equation (24) is an ODE with a velocity field that coincides
with the velocity field in equation (9).

##  Appendix C Flow Matching Sampling with Guidance for Offline RL

Comparing with standard generative modeling, the sequence model trained for RL
needs to condition on the current state $s_{t}$, see Section 5. Therefore, the
sampling process is slightly different from Algorithm 2, as we need to zero
out the vector fields corresponding to $s_{t}$, as shown in Algorithm 4.
Sampling for the goal-conditioned model can be done similarly.

Input: initial state $s_{0}$, target return $R$, guidance parameter $\omega$,
standard deviation of the starting distribution $\sigma$, number of ODE steps
$n_{\text{ode}}$

Sample $x_{0}\sim\mathcal{N}(0,\sigma^{2}I)$

$h\leftarrow 1/n_{\text{ode}}$ $\triangleright$ step size

for  _$t=0,h,\ldots,1-h$_ do

$x_{t}[0]\leftarrow s_{0}$ $\triangleright$ fix the known token

$\tilde{u}_{t}(\cdot)\leftarrow(1-\omega)u^{\theta}_{t}(\cdot)+\omega
u^{\theta}_{t}(\cdot|R)$ $\triangleright$ compute the velocity field under
guidance

Define $\bar{u}_{t}(\cdot):\mathbb{R}^{p}\mapsto\mathbb{R}^{p}$ such that for
any $x\in\mathbb{R}^{p}$, the first dimension of $\bar{u}_{t}(x)$ is equal to
$0$, and the other dimensions are the same as $\tilde{u}_{t}(x)$
$\triangleright$ zero out the VF for the known token

$x_{t+h}\leftarrow$ ODEStep$(\bar{u}_{t},x_{t})$

Output: $x_{1}$

Algorithm 4 Flow Matching Sampling with Guidance for Offline RL

##  Appendix D Experiment Details

We summarize the architecture and other hyperparameters used for our
experiments. For all the experiments, we use our own PyTorch implementation
that is heavily influenced by the following codebases: Decision Diffuser
https://github.com/anuragajay/decision-diffuser
Diffuser https://github.com/jannerm/diffuser/

We train both guided flows and guided diffusion models for state sequences of
length $H=64$. The probability of null conditioning $p_{\text{uncond}}$ is set
to $0.25$. The batch size is $64$. We normalize the discounted RTG by a task-
specific reward scale, which is $400$ for hopper, $550$ for walker and $1200$
for halfcheetah. The final model parameter $\bar{\theta}$ we consider is an
exponential moving average (EMA) of the obtained parameters over the course of
training. For every $10$ iteration, we update
$\bar{\theta}=\beta\bar{\theta}+(1-\beta)\theta$, where the exponential decay
parameter $\beta=0.995$. We train the sequence model for $2\times 10^{6}$
iterations, and checkpoint the EMA model every $200$k iteration.

#### Guided Flows

We use a temporal U-net to model the velocity field $u^{\theta}$. It consists
of 6 repeated residual blocks, where each block consists of 2 temporal
convolutions followed by the group norm Wu and He (2018) and a final Mish
nonlinearity activation Mish (2019). The time $t$ is first trainsformed to its
sinusoidal position encoding and projected to a latent space via a 2-layer
MLP, and the RTG $g(\bm{s})$ is transformed into its latent embedding via a
3-layer MLP. The model is optimized by the Adam optimzier Kingma and Ba
(2014). The learning rate is $2\times 10^{-4}$ for hopper-medium-expert,
$3\times 10^{-4}$ for walker-medium-replay and $10^{-4}$ for all the other
datasets.

#### Guided Diffusion Models

We use a temporal U-net to model the noise ${\epsilon}_{\theta}$, with the
same architecture used for guided flows. The model is also optimized by the
Adam optimzier, where the learning rate $2\times 10^{-4}$ for all the
datasets.

#### Inverse Dynamics Model

The inverse dynamics model is modeled by an MLP with 2 hidden layers, 1024
hidden units per layer, and a $10\%$ dropout rate. We use the Adam optimizer
with learning rate $10^{-4}$. We randomly sample 10% of offline trajectories
as the validation set. We train the IDM for 100k iterations, and use the one
that yields the best validation performance.

##  Appendix E Additional Experiments

###  E.1 Computational Speed Comparison

Figure E.1: The CPU time of generating a batch of $704$-dimensional vectors
(The subsequence length is $64$, and each state is of dimension $11$), using
different numbers of internal steps. The time consumed by the diffusion model
and the flow are roughly the same.

###  E.2 Flow Generated State Sequences

|
---|---
|
|
|
Figure E.2: Comparison of true state sequences and flow generated state
sequences. In each panel, the top row plots a randomly sampled true state
sequence from the hopper-medium dataset, and the bottom row plots the sequence
sampled from the flow, conditioning on the first state and a large out-of-
distribution RTG. The guidance weight is $3.0$.