# Hippocampal representations emerge when training recurrent neural networks
on a memory dependent maze navigation task

Justin Jude*, Matthias H. Hennig**
Institute for Adaptive and Neural Computation, School of Informatics,
University of Edinburgh, United Kingdom
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Can neural networks learn goal-directed behaviour using similar strategies to
the brain, by combining the relationships between the current state of the
organism and the consequences of future actions? Recent work has shown that
recurrent neural networks trained on goal based tasks can develop
representations resembling those found in the brain, entorhinal cortex grid
cells, for instance. Here we explore the evolution of the dynamics of their
internal representations and compare this with experimental data. We observe
that once a recurrent network is trained to learn the structure of its
environment solely based on sensory prediction, an attractor based landscape
forms in the network’s representation, which parallels hippocampal place cells
in structure and function. Next, we extend the predictive objective to include
Q-learning for a reward task, where rewarding actions are dependent on delayed
cue modulation. Mirroring experimental findings in hippocampus recordings in
rodents performing the same task, this training paradigm causes nonlocal
neural activity to sweep forward in space at decision points, anticipating the
future path to a rewarded location. Moreover, prevalent choice and cue-
selective neurons form in this network, again recapitulating experimental
findings. Together, these results indicate that combining predictive,
unsupervised learning of the structure of an environment with reinforcement
learning can help understand the formation of hippocampus-like representations
containing both spatial and task-relevant information.

##  1 Introduction

Recurrent neural networks have been used to perform spatial navigation tasks
and the subsequent study of their internal representations has yielded
dynamics and structures that are strikingly biological. Metric (Cueva and
Wei,, 2018; Banino et al.,, 2018) and non-metric (Recanatesi et al.,, 2019)
representations mimicking grid (Fyhn et al.,, 2004) and place cells (O’Keefe
and Nadel,, 1978) respectively form once the recurrent network has learned a
predictive task in the context of a complex environment. Cueva et al., (2020)
demonstrates not only the emergence of characteristic neural representations,
but also hallmarks of head direction system cells when training a recurrent
network on a simple angular velocity integration task. Biologically, non-
metric representations are associated with landmark spatial memory, in which
place cells within the mammalian hippocampus fire when the associated organism
is present in a corresponding place field. Extrafield firing of place cells
occurs when these neurons spike outside of these contiguous place field
regions. Here we show that recurrent neural networks (RNNs) not only form
corresponding attractor landscapes, but also produce representations with
internal dynamics that closely resemble those found experimentally in the
hippocampus when performing goal-directed behaviour.

Research in neuroscience such as that of Johnson and Redish, (2007), shows
that spatial representations in mice in the CA3 region of the hippocampus
frequently fire nonlocally. Griffin et al., (2007) show that a far higher
proportion of hippocampal neurons in the CA1 region in rats performing an
episodic task in a T-shaped maze encode the phase of the task rather than
spatial information (in this case trajectory direction). Ainge et al., (2007)
show CA1 place cells encode destination location at the start position of a
maze. Lee et al., (2006) demonstrate that place fields of CA1 neurons
gradually drift toward reward locations throughout reward training on a
T-shaped maze.

In this work we show that a recurrent neural network learning a choice-reward
based task using reinforcement learning, in conjunction with predictive
sensory learning in a T-shaped maze produces an internal representation with
consistent extrafield firing associated with consequential decision points. In
addition we find that the network’s representation, once trained, follows a
forward sweeping pattern as identified by Johnson and Redish, (2007). We then
show that a higher proportion of units in the trained network show strong
selectivity for the encoding or choice phase of the task than the proportion
showing selectivity for spatial topology. Importantly, these properties only
emerge during predictive learning, where task learning is much faster compared
to traditional deep Q learning.

##  2 Method

Figure 1: Left, the wall observation and cue received by the network at each
timestep. Right, the entangled predictive task the LSTM network is pre-trained
on in order to generate a non-metric map of the maze environment.

We use a form of the cued-choice maze used by Johnson and Redish, (2007) which
has a central T structure with returning arms, shown in Figure 1. All walls of
the maze are tiled with distinct RGB colours which are generated at random and
remain fixed throughout. An agent is initially learning to predict the next
sensory stimulus given its movement. This combination of unsupervised learning
and exploration has been shown previously to produce place cell-like encoding
of the agent’s position (Recanatesi et al.,, 2019). Next, rewards at four
possible locations are introduced and the agent is tasked with associating a
cue with the rewarding trajectory. The agent has four vision sensors, one in
each cardinal direction, reading the wall RGB colours they intersect. The cue
tone is played to the agent as it passes the halfway point of the central maze
stem. A low frequency cue indicates that the agent will turn left at the top
of the maze stem and a high frequency cue indicates a right turn. These cue
tones take the form of a high or low valued scalar perturbed with normally
distributed noise if at a cue point, with a zero value given at all other
locations. These four RGB colours as well as the cue frequency at the current
location make up the total input received by the agent.

The agent is controlled by a recurrent neural network comprised of a 380 unit
Long-Short term memory (Hochreiter and Schmidhuber,, 1997) (LSTM) network with
a single layered readout for the prediction of RGB values. We first pre-train
the network by tasking it with predicting the subsequent observation of wall
colours from the currently observable wall colours given its trajectory
through the maze. The agent’s starting location is at the bottom of the
central stem of the T maze and a trajectory of left or right at the top of the
central stem is chosen pseudorandomly, depicted with red and blue arrows
respectively in Figure 1 and corresponding to the low (red trajectory) or high
(blue trajectory) cue tone value given halfway up the stem. As in the
experiments by Johnson and Redish, (2007), during pre-training the agent does
not choose any of its actions and is only learning to predict the sequence of
wall colors it encounters. In a given pre-training iteration, we collect all
observations as the agent traverses the maze until it returns to the start
location at the bottom of the central stem and finally train the LSTM on the
entire collected trajectory. The network is trained with a mean-squared error
loss of predicted and target wall colours (Eq. 1), with model parameters
optimised using Adam (Kingma and Ba,, 2015) and a learning rate of 0.001.

$loss_{rgb}=\frac{1}{n}\sum_{i=1}^{n}(y_{rgb}-(W_{rgb}h_{t}+b_{rgb}))^{2}$ (1)

To solve this task, the network has to maintain the cue tone played in its
internal memory for several time steps in order to predict subsequent wall
colours from the top of the central stem. In our model, this is achieved
through the network forming a non-metric representation (attractor landscape)
of the maze environment, as also demonstrated by Xu and Barak, (2020).
Similarly, behavioural experiments typically have a comparable familiarisation
phase with the environment before reward-based tasks are introduced (Johnson
and Redish,, 2007; Griffin et al.,, 2007).

Figure 2: For the joint task to be learned by the LSTM network, we introduce
secondary cue points, where the same cue tone as that played at the primary
cue point will be repeated if and only if the agent has proceeded in turning
in the direction corresponding to the cue tone frequency given at the primary
cue location. The agent is free to choose the next action to be taken when
traversing the maze at either the choice point at the top of the stem of the
maze or at the secondary cue locations. There are two potential reward sites
on both returning arms, with the reward sites being active if the agent is on
the returning arm corresponding to the cue tone frequency.

Once the LSTM has formed an internal representation of the maze, the agent is
tasked with navigating towards potential reward sites whose location is
indicated by the cue signal: a low frequency cue indicates active reward sites
on the left return arm and a high frequency cue indicates active reward sites
on the right return arm - the cue tone and corresponding side of active reward
sites are together chosen randomly at each iteration with a secondary cue
given if the agent has turned correctly. In this phase there are three choice
points wherein the agent is able to choose its next action and is constrained
to follow the forward maze direction elsewhere: at the top of the maze stem
and at the two secondary choice points (Figure 2), with initially random
movement at these points during reward training. There are 5 steps between the
cue and choice points and 7 steps from the choice point to the first reward
site on either return arm. The inclusion of the secondary cues as additional
choice points was motivated by the experimental set up used by Johnson and
Redish, (2007), to compare the network activity at these points to
experimental data. These secondary points also give the agent the opportunity
to backtrack on its decision made at the primary choice point in light of
further environmental observation (the presentation or lack thereof of the
secondary cue), and make learning more efficient in our model. This may
explain how it speeds up training the animals in the same task.

We additionally introduce a new single layered readout for the LSTM network
which predicts state-action values associated with the four cardinal
directions in relation to the agent’s current position and direction. At each
timestep, this ensemble receives the agent’s environment observation and the
agent follows an epsilon-greedy policy (starting with fully random movement at
choice points and a decaying epsilon thereafter) for choosing optimal actions
of those available at each of the three choice points.

The recurrent network controlling the agent is trained on a weighted combined
loss of a reinforcement learning (RL) task loss and the previously described
predictive wall colour loss:

$loss_{combined}=|Q(s,a)-(r+\gamma\cdot
Q^{\prime}(s^{\prime},\operatorname*{arg\,max}_{a^{\prime}}Q(s^{\prime},a^{\prime}))|+\lambda\cdot
loss_{rgb}$ (2)

The first component of this loss is the difference between predicted and
observed state-action values which are represented by Q-values (Watkins and
Dayan,, 1992), which are a prediction of future global reward:

$Q(s,a)=W_{Q}h_{t}+b_{Q}$ (3)

We use double-Q learning (Van Hasselt et al.,, 2016) to train the agent on the
task, updating the target Q value predictor ($Q^{\prime}$ \- a LSTM with same
number of units) every 15 training iterations. Double-Q learning allows for
optimal performance on the reward task in drastically fewer agent maze
traversals and network training iterations than with standard DQN (Mnih et
al.,, 2013) based Q-learning which suffers from overestimation of Q-values. We
settle on a discount factor ($\gamma$) of 0.8 as values higher than this
regularly cause the network to converge on solutions wherein the agent does
not take the most direct path to reward locations, with backtracking at
secondary choice points. The second loss component is the sensory prediction
task which we used to pre-train the network ($\lambda=0.5$). After the network
has been pre-trained (optimising to minimise Eq. 1), the network achieves
perfect performance on the predictive task. This loss component is included
when training the network on the reward task so that the spatial map (non-
metric attractor landscape) of the maze environment formed during pre-training
is maintained throughout Q-learning. This ensures the map is not overwritten
as would happen when Q-learning is performed alone, and leads to faster task
learning (see results). We optimise the network for this joint task using Adam
and a learning rate of 0.0005, which we find improves the rate of convergence
with optimal task performance, as opposed to higher learning rates which still
converge but with backtracking often inherent in task solutions.

In contrast to much of the previous work on spatial representations in
recurrent networks, we do not give the network any indication of the agent’s
location or movement. This makes the task considerably more difficult due to
the unpredictable movement possible at choice points during the reward task.
The network is coerced into storing the current movement direction of the
agent in its representation, in addition to storing the cue frequency. As
such, a network of Gated Recurrent Units (Cho et al.,, 2014) (GRUs) or vanilla
RNN units was unable to perform well in either the pre-training or joint RL
task due to these prevalent long term dependencies (18 steps between cue and
final reward).

To analyse the representations formed by the network, we train a further
single layered fully connected network with a softmax layer (shown in green in
Figure 2) to predict the agent’s next location using the activity of the LSTM.
There is no backpropagation of gradients between this predictor and the LSTM
network, and the predictor is trained at the end of reward training. The plots
in Figures 5 and 6 are distributions indicating the probability of agent
location inferred from LSTM activity by the predictor. This is used in place
of the decoding algorithm used by Johnson and Redish, (2007) to predict the
neuronally inferred maze location of rats when performing a cue based task.

##  3 Results

The agent learns the sensory prediction task to a high degree of recall and
after around a thousand training iterations (combined loss with pre-training
in Figure 3), the agent was able to achieve perfect performance on the reward
task when the LSTM network had 380 or more units (Fig. 3, right). We trained
the reinforcement learning (Eq. 2) portion of the task in an epsilon greedy
manner, with a steadily decaying epsilon to ensure that the agent would choose
the rewarding path consistently once actions were chosen at choice points
completely by the network. Notably, the agent did not turn at either of the
secondary choice points once training had completed - only at the primary
choice point.

We attempted to run the reinforcement learning task alone in a maze with no
sensory input except the reward cue. In this scenario the network is not able
to learn the task due to a lack of self-localisation and is unable to perform
the task based on step counting between the cue and choice point. In addition,
the reward based reinforcement learning task was attempted using Q-learning
alone with a loss function that did not include the wall colour prediction
error, both with and without pre-training (shown in Fig. 3, left). In both
cases we find that the reward task is not learnable with the same higher rate
of epsilon decay we use for the combined loss function with pre-training, as
the network quickly forgets the attractor landscape of the maze formed during
pre-training, which we maintain through the combined loss (Eq. 2). We also
find the network can solve the reward task using the combined loss without
pre-training, albeit in around 3 times the number of maze traversals as with
the use of the spatial map formed in the pre-trained case (Fig. 3, left).

Figure 3: Left: Success rate (proportion of direct traversals to reward
locations) of each set of training paradigms on the reward task, averaged over
10 initial conditions and random wall colours using optimal rate of epsilon
decay for each paradigm, each shown with a 95% confidence interval. Attractor
landscape formed during pre-training alongside combined loss allows network to
achieve perfect performance on reward task in relatively few maze traversals.
Q-learning alone without pre-training also achieves perfect performance in
more than twice the number of maze traversals. Q-learning alone with pre-
training takes far more maze traversals to converge (and is less likely to be
optimal) due to the non-random initial state of network and inability to
utilise the spatial map formed. Combined training without pre-training also
takes relatively many maze traversals to converge due to a relatively
difficult joint task with no biased initial state. Right: Pre-trained network
optimised with combined loss converges at similar rates with different network
sizes above 380 units.

###  3.1 Extrafield place cell firing

First, we investigate the representation learned by the the network during
these two stages of training. Pre-training causes the formation of discrete
attractors that resemble place cells in the hippocampus. Individual units in
the network generally have well isolated place fields, which together cover
the whole maze and therefore allow reliable decoding of agent location. In
addition to an increase in activity in a particular unit when the agent moves
across its respective place field, we also observe substantial extrafield
firing of these units. This activity occurs mainly at the primary cue location
and at the first choice point after pre-training. After training on the reward
task, in addition to the place fields, the network also has units with
extrafield activity at the secondary choice points (Fig. 4E).

In the top row of Figure 3(A-D) we show activity in 4 reward trained LSTM
units obtained through the collection of unit activity from a full left sided
trajectory from the maze start point returning to the start point with cues
presented, together with a full right sided trajectory. We show all activity
from this activity collection in the top row of Figure 3(A-D) and proceed to
outline the maze areas for each unit with activity higher than 30% of the peak
activity of that particular unit (mirroring the experimental threshold used by
Johnson and Redish, (2007)), denoting them as place fields corresponding to
these LSTM units. In experiments, rodents seem to pause at high consequence
decision points (Johnson and Redish,, 2007) with alternating head movement
behaviour signifying vicarious trial and error (VTE) (Muenzinger,, 1938; Hu
and Amsel,, 1995). In the activity plots in the bottom row of Figure 3(A-D),
we simulate this using our reward trained model by running the agent from the
start position at the bottom of the maze stem, then pausing it at the top of
the stem, with a left cue presented halfway up. We show activity above 60% of
unit peak activity (identified with the previously collected aggregated
activity) shown in addition to the previously identified place fields.

The network representation seems to sample both return arms, with surprisingly
high extrafield activity in the shown LSTM units when the agent is paused at
the maze choice point, a location for which these units do not usually have
corresponding activity (Fig. 4A-D). We define nonlocal firing as unit activity
above 60% of peak averaged unit activity when running the agent along the
central stem (bottom row Fig. 4A-D) to observe only the most poignant
extrafield behaviour.

Figure 4: A-D) Top row: Activity maps showing well isolated place fields of
four LSTM units (acting as place cells) indicated in dotted regions after the
reward task. Place fields determined by contiguous locality with average
activity exceeding 30% peak unit activity during a single left trajectory
followed by a right trajectory. Bottom row: LSTM unit activity exceeding 60%
of previously averaged peak unit activity for the given neuron when agent run
from bottom of maze stem to top of stem and given a low frequency (left) cue
tone halfway up the stem, then stationary at choice point with LSTM network
repeatedly receiving observation from choice point for timesteps thereafter
(shown in addition to previously determined unit place fields in dotted
regions). A, B) Strong extrafield firing contiguously from cue to choice
point. C, D) High extrafield firing at choice point while agent is paused at
top of stem. E) Place fields (determined from average activity on both
trajectories) of four LSTM units outlined in dotted areas after reward based
task. High levels of consistent extrafield firing at primary and secondary cue
points in 56% of LSTM units.

###  3.2 Forward moving representation

The internal dynamics of the LSTM network has an inherently forward looking
representation of the maze once pre-trained in a predictive manner. As
depicted in Fig. 5, whilst the agent is stationary, the dynamics of the LSTM
network moves forward through the maze, incorporating the trajectory
modulation of the cue played halfway up the maze stem. The forward movement of
the representation is also notable for having an inconsistent velocity, where
the LSTM inferred agent location jumps (Hasselmo,, 2009) from the top of the
maze to lower down the arm (timestep 16 in Fig. 5).

Figure 5: LSTM inferred agent position after pre-training on maze. The agent
is run from the start at timestep 1 to timestep 4 where it receives a low
frequency cue (indicating a left turn). At timestep 9 the agent is stopped at
the top of the maze stem and the LSTM is given the environment observation
from this location for the remainder of the shown timesteps. The inferred
position then moves left according to the cue with the position seeming to
jump abruptly between timesteps 15 and 16. The inferred position then moves
back to the starting position at timestep 26. We observe an analogous inferred
forward moving representation on the right side of the maze with a high
frequency cue. Figure 6: LSTM representation after reward training. As
previously, we run the agent from the start position to the top of the stem of
the maze at timestep 9 with a low frequency (left) cue tone at timestep 4.
Again, the agent is stopped at this position with the LSTM network receiving
the environment observation from this position for the remainder of the shown
timesteps. As with the network purely trained on the predictive task, the
representation moves in the direction corresponding to the frequency of the
given cue tone. Then between timesteps 14 and 15, the inferred position jumps
from the return arm with active reward sites to the alternate arm, with the
inferred position moving from this position to the start location fairly
consistently. Then the inferred position jumps again at timestep 32 to the
rewarding return arm and moves constantly to the start position.

In stark contrast to the dynamics of the LSTM network after predictive pre-
training, following training on the reward task the forward representation of
the LSTM is still looking ahead of the agent but is now displaying sweeping
behaviour (Fig. 6) which is identified experimentally in rats by Johnson and
Redish, (2007) when performing cue based tasks. When the agent is stationary
at the choice point, we observe the representation moving ahead of the agent -
first in the direction corresponding to the cue given at the first cue point
and then abruptly down the opposing arm of the maze towards the starting
location, thereafter the representation moves down the correct arm
(corresponding to the cue) and becomes stationary at the maze start location.
This path switching behaviour is reliably observed in networks trained on the
combined loss (Eq. 2) with and without pre-training, with differing numbers of
units and initial conditions as long as the reward task is solved without
backtracking at secondary cue locations. The network lacks a sweeping or
forward moving representation when trained on the reward task with Q-learning
alone, regardless of pre-training. Thus pre-training does not contribute to
sweeping or path switching behaviour.

Figure 7: UMAP manifold of LSTM network dynamics of complete left trajectory
(dark blue) and complete right trajectory (red) shown along with manifold of
dynamics when agent run from start location to choice point with left cue
(light blue) and right cue (pink) given at cue point and agent paused in place
at the top of the maze stem. A few timesteps after the agent is paused, the
dynamics of the left cue paused agent (light blue) switches manifold path
abruptly from running alongside the complete left trajectory path (blue) and
joins the right trajectory path (red), following this for many timesteps
before ultimately resulting at the same manifold end position as the complete
left trajectory manifold path (blue). This is analogous for the right cue
paths (red and pink).

We further investigate the network representation using Uniform Manifold
Approximation and Projection (UMAP) (McInnes et al.,, 2018). Figure 7 shows
generally connected manifolds, with closer inspection revealing the dynamics
which leads to the sweeping arm behaviour in Figure 6 when the agent is
stationary at the primary choice point. Zeroing visual input while the agent
is paused at the choice point gives comparable representation dynamics to that
observed in Figures 6 and 7.

Separately, we find that place fields of particular LSTM units drift forwards
from their original firing positions after pre-training, towards the reward
locations on the return arms throughout reward training, as shown
experimentally in CA1 neurons in Lee et al., (2006). We observe this behaviour
in 50 out of 380 network units (13%), with final resting locations of place
fields at reward locations (seen in Appendix Figure 10). This is possibly
explained by the gradient of Q values (prediction of predicted reward)
spreading backwards from reward locations (Hasselmo,, 2005) and becoming
stronger throughout training.

###  3.3 Selectivity of neuronal units

In addition to a forward sweeping representation, this trained network also
exhibits neural selectivity that closely matches hippocampal circuits. Griffin
et al., (2007) reported that after reward learning, hippocampal neurons were
more strongly selective for the encoding or choice phase of a task rather than
the direction of the organism’s trajectory. We garner the preference of
selectivity of each neuronal unit in our network using a discrimination index
used by Griffin et al., (2007) for the turn direction selectivity
($DI_{\text{turn}}$) and the phase selectivity ($DI_{\text{phase}}$):

$DI_{\text{turn}}=\frac{FR_{\text{right}}-FR_{\text{left}}}{FR_{\text{right}}+FR_{\text{left}}}\quad
DI_{\text{phase}}=\frac{FR_{\text{cue}}-FR_{\text{choice}}}{FR_{\text{cue}}+FR_{\text{choice}}}$
(4)

where $FR_{\text{right}}$ for a particular LSTM unit is the mean firing rate
from the cue point on the central stem to the choice point at the top of the
stem on trajectories where the agent turns right at the choice point.
Similarly $FR_{\text{left}}$ is the mean stem firing rate when the agent turns
left. $FR_{\text{cue}}$ is the firing rate at the cue (encoding) point
averaged over both left and right trajectories and similarly
$FR_{\text{choice}}$ is the averaged firing rate at the choice (sampling)
point.

Figure 8: Histograms showing LSTM unit discrimination index for turn direction
selectivity ($DI_{\text{turn}}$) vs task phase selectivity
($DI_{\text{phase}}$). A highly negative selectivity index for turn direction
indicates a neuronal unit which exhibits high levels of selectivity (uniquely
high network activity) for a leftward trajectory and a highly positive
selectivity index indicates selectivity for a rightward trajectory. A negative
selectivity for task phase indicates a neuron which is highly selective for
the choice (retrieval) phase of the goal based task whereas a positive index
indicates a neuron which is highly selective for the cue (encoding) phase of
the task.

The firing areas used for selectivity measurement are insets in Figure 8. We
use the stem above the cue point to assess turn direction selectivity, and the
cue/choice points to assess encoding and sampling ($DI_{\text{phase}}$).
Figure 8 shows a higher proportion of LSTM units are strongly task selective
rather than turn selective, with significantly more units having large
absolute $DI_{\text{phase}}$ indices than $DI_{\text{turn}}$ indices.

In addition, the reward trained network is found to have a disproportionately
high number of units (163 out of 380 LSTM units) with place fields at the
start location of the maze. Moreover, we find evidence of conditional
destination encoding in these units which were heavily differentiated in their
firing with respect to particular rewarding locations, as shown experimentally
in CA1 hippocampal place cells (Ainge et al.,, 2007; Wood et al.,, 2000;
Ferbinteanu and Shapiro,, 2003). 59.5% of units with a place field at the maze
start location fired uniquely at this point for rewarding locations on a
particular return arm.

##  4 Discussion

In this work we show that networks trained with a combined predictive and
goal-based objective exhibit functional dynamics and selectivity behaviour
coinciding with that of hippocampal neurons. We demonstrate that extrafield
firing activity of network units emerge when a simulated agent, which is
trained on a goal based reward task in a T-shaped maze, pauses at decision
points - suggesting intrinsic dynamics are encoding the future trajectory of
the agent. This mirrors experimental results in hippocampal place cells in
rats (Johnson and Redish,, 2007; Frank et al.,, 2000). At the same time, we
find that networks using this combined objective, following pre-training only
on a sensory prediction task, can learn the correct goal-directed behaviour
much faster than an equivalent network with only a Q learning objective.

Previous work shows that metric neural representations of environments form
when an RNN is optimised to predict agent position from agent velocity (Cueva
and Wei,, 2018; Banino et al.,, 2018) and non-metric representations form when
an RNN is trained to predict future sensory events given direction of movement
(Recanatesi et al.,, 2019). When training our model we do not provide the LSTM
network with any explicit information about location or direction, it only
receives sensory information. This is similar to the purely contextual input
received by the model pre-trained by Xu and Barak, (2020) where no velocity
input is given, however, the network used by these authors is still trained on
position and landmark prediction in a supervised way.

Instead, our training paradigm forces the LSTM to maintain an implicit notion
of movement within its internal state in relation to environmental
observations. This, in conjunction with the consideration that model-free RL
methods such as Q-learning perform poorly on tasks in dynamic environments
such as ours (Dolan and Dayan,, 2013), and the long term dependency on the
delayed cue in perspective of the choice location, makes the task outlined in
Figure 2 particularly challenging.

Training on a sensory predictive task causes the formation of a non-metric
place cell like representation in the activations of network units, similarly
to Recanatesi et al., (2019). These units demonstrate nonlocal extrafield
firing (Appendix Figure 9) and after reward training (Figure 4). Johnson and
Redish, (2007) find that this extrafield firing is particularly striking at
consequential decision points where rats usually pause in order to sample
previously seen trajectories. We observe that cue or choice point extrafield
activity is evident in most LSTM units after training on the reward task. This
is likely due to the increased precedence these points have in the agent
reaching reward locations. Together the trained LSTM network units form a
representation which sweeps along the paths available to the agent, first down
the reward path and then the other, as shown in Figure 6 and demonstrated in
rats in Johnson and Redish, (2007).

Although hippocampal place cells are critical for spatial memory (Nakazawa et
al.,, 2002; Florian and Roullet,, 2004; Sandi et al.,, 2003; Redish and
Touretzky,, 1998; Miller et al.,, 2020), it is currently unclear by what
mechanism an ensemble of place cells contributes to a representation of goal-
directed behaviour (Morris,, 1990). Our model and training paradigm is in
keeping with the hypothesis that the hippocampus is involved in maintaining a
conjunctive representation of cognitive maps and sensory information
(Whittington et al.,, 2019). We show that this paradigm can be extended with
predictive learning of Q-values of anticipated future reward, and show that
the resulting representation is well suited for learning actions leading from
a cue to a reward. Importantly, this representation emerges solely from
sampling sensory inputs and predicted rewards, while reinforcement learning
itself remains model-free and is initially random. The surprising similarity
of the task-dependent activity in our simulations and experimentally recorded
neural activity in similar tasks suggests that the model may replicate central
aspects of learning and planning in the hippocampus. Our trained model could
improve understanding of hippocampal function by testing hypotheses regarding
previously unobserved dynamics inexpensively. This could be performed on maze
environments such as this work, or more open arena settings once the model is
retrained.

## References

  * Ainge et al., (2007) Ainge, J. A., Tamosiunaite, M., Woergoetter, F., and Dudchenko, P. A. (2007).  Hippocampal CA1 place cells encode intended destination on a maze with multiple choice points.  Journal of Neuroscience, 27(36).
  * Banino et al., (2018) Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., Pritzel, A., Chadwick, M. J., Degris, T., Modayil, J., Wayne, G., Soyer, H., Viola, F., Zhang, B., Goroshin, R., Rabinowitz, N., Pascanu, R., Beattie, C., Petersen, S., Sadik, A., Gaffney, S., King, H., Kavukcuoglu, K., Hassabis, D., Hadsell, R., and Kumaran, D. (2018).  Vector-based navigation using grid-like representations in artificial agents.  Nature.
  * Cho et al., (2014) Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014).  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  In EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference.
  * Cueva et al., (2020) Cueva, C. J., Wang, P. Y., Chin, M., and Wei, X.-X. (2020).  Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks.  In International Conference on Learning Representations.
  * Cueva and Wei, (2018) Cueva, C. J. and Wei, X. X. (2018).  Emergence of grid-like representations by training recurrent neural networks to perform spatial localization.  In 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings. International Conference on Learning Representations, ICLR.
  * Dolan and Dayan, (2013) Dolan, R. J. and Dayan, P. (2013).  Goals and habits in the brain.
  * Ferbinteanu and Shapiro, (2003) Ferbinteanu, J. and Shapiro, M. L. (2003).  Prospective and retrospective memory coding in the hippocampus.  Neuron, 40(6).
  * Florian and Roullet, (2004) Florian, C. and Roullet, P. (2004).  Hippocampal CA3-region is crucial for acquisition and memory consolidation in Morris water maze task in mice.  Behavioural Brain Research, 154(2).
  * Frank et al., (2000) Frank, L. M., Brown, E. N., and Wilson, M. (2000).  Trajectory encoding in the hippocampus and entorhinal cortex.  Neuron, 27(1).
  * Fyhn et al., (2004) Fyhn, M., Molden, S., Witter, M. P., Moser, E. I., and Moser, M. B. (2004).  Spatial representation in the entorhinal cortex.  Science.
  * Griffin et al., (2007) Griffin, A. L., Eichenbaum, H., and Hasselmo, M. E. (2007).  Spatial representations of hippocampal CA1 neurons are modulated by behavioral context in a hippocampus-dependent memory task.  Journal of Neuroscience, 27(9).
  * Hasselmo, (2005) Hasselmo, M. E. (2005).  A model of prefrontal cortical mechanisms for goal-directed behavior.  Journal of Cognitive Neuroscience, 17(7).
  * Hasselmo, (2009) Hasselmo, M. E. (2009).  A model of episodic memory: Mental time travel along encoded trajectories using grid cells.  Neurobiology of Learning and Memory, 92(4).
  * Hochreiter and Schmidhuber, (1997) Hochreiter, S. and Schmidhuber, J. (1997).  Long Short-Term Memory.  Neural Computation.
  * Hu and Amsel, (1995) Hu, D. and Amsel, A. (1995).  A simple test of the vicarious trial-and-error hypothesis of hippocampal function.  Proceedings of the National Academy of Sciences of the United States of America, 92(12).
  * Johnson and Redish, (2007) Johnson, A. and Redish, A. D. (2007).  Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point.  Journal of Neuroscience, 27(45).
  * Kingma and Ba, (2015) Kingma, D. P. and Ba, J. L. (2015).  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.
  * Lee et al., (2006) Lee, I., Griffin, A. L., Zilli, E. A., Eichenbaum, H., and Hasselmo, M. E. (2006).  Gradual Translocation of Spatial Correlates of Neuronal Firing in the Hippocampus toward Prospective Reward Locations.  Neuron, 51(5):639–650.
  * McInnes et al., (2018) McInnes, L., Healy, J., Saul, N., and Großberger, L. (2018).  UMAP: Uniform Manifold Approximation and Projection.  Journal of Open Source Software, 3(29).
  * Miller et al., (2020) Miller, T. D., Chong, T. T., Davies, A. M., Johnson, M. R., Irani, S. R., Husain, M., Ng, T. W., Jacob, S., Maddison, P., Kennard, C., Gowland, P. A., and Rosenthal, C. R. (2020).  Human hippocampal CA3 damage disrupts both recent and remote episodic memories.  eLife, 9.
  * Mnih et al., (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013).  Playing atari with deep reinforcement learning.
  * Morris, (1990) Morris, R. (1990).  Does the hippocampus play a disproportionate role in spatial memory.  Discussions in Neuroscience, 6:39–45.
  * Muenzinger, (1938) Muenzinger, K. F. (1938).  Vicarious Trial and Error at a Point of Choice: I. A General Survey of its Relation to Learning Efficiency.  Pedagogical Seminary and Journal of Genetic Psychology, 53(1).
  * Nakazawa et al., (2002) Nakazawa, K., Quirk, M. C., Chitwood, R. A., Watanabe, M., Yeckel, M. F., Sun, L. D., Kato, A., Carr, C. A., Johnston, D., Wilson, M. A., and Tonegawa, S. (2002).  Requirement for hippocampal CA3 NMDA receptors in associative memory recall.  Science, 297(5579).
  * O’Keefe and Nadel, (1978) O’Keefe, J. and Nadel, L. (1978).  The hippocampus as a cognitive map.  Clarendon Press, Oxford, United Kingdom.
  * Recanatesi et al., (2019) Recanatesi, S., Farrell, M., Lajoie, G., Deneve, S., Rigotti, M., and Shea-Brown, E. (2019).  Signatures of low-dimensional neural predictive manifolds.  Cosyne Abstracts 2019, Lisbon, PT.
  * Redish and Touretzky, (1998) Redish, A. D. and Touretzky, D. S. (1998).  The Role of the Hippocampus in Solving the Morris Water Maze.  Neural Computation, 10(1).
  * Sandi et al., (2003) Sandi, C., Davies, H. A., Cordero, M. I., Rodriguez, J. J., Popov, V. I., and Stewart, M. G. (2003).  Rapid reversal of stress induced loss of synapses in CA3 of rat hippocampus following water maze training.  European Journal of Neuroscience, 17(11).
  * Van Hasselt et al., (2016) Van Hasselt, H., Guez, A., and Silver, D. (2016).  Deep reinforcement learning with double Q-Learning.  In 30th AAAI Conference on Artificial Intelligence, AAAI 2016.
  * Watkins and Dayan, (1992) Watkins, C. J. C. H. and Dayan, P. (1992).  Q-learning.  Machine Learning, 8(3-4).
  * Whittington et al., (2019) Whittington, J. C., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., and Behrens, T. E. (2019).  The Tolman-Eichenbaum Machine: Unifying space and relational memory through generalisation in the hippocampal formation.  bioRxiv, page 770495.
  * Wood et al., (2000) Wood, E. R., Dudchenko, P. A., Robitsek, R. J., and Eichenbaum, H. (2000).  Hippocampal neurons encode information about different types of memory episodes occurring in the same location.  Neuron, 27(3).
  * Xu and Barak, (2020) Xu, T. and Barak, O. (2020).  Implementing inductive bias for different navigation tasks through diverse rnn attrractors.  In International Conference on Learning Representations.

##  Appendix A Appendix

###  A.1 Extrafield place cell firing after sensory prediction task

After pre-training on the sensory prediction task outlined in Figure 1, we
observe that when the agent is paused at the top of the stem of the maze the
network representation moves far ahead of the agent, caused by extrafield
activity of many neurons in the network.

Figure 9: A, B, C) Top row: well isolated place fields of three LSTM units
indicated in dotted regions after pre-training. Place fields determined by
contiguous locality with average activity exceeding 30% peak field activity
during a single left trajectory followed by a right trajectory. Bottom row:
agent run from bottom of maze stem to top of stem (and given a low frequency
cue tone halfway up the stem) and paused at choice point with LSTM network
repeatedly receiving observations from choice point for timesteps thereafter.
A) Strong extrafield firing at choice point with some activity at the cue
point. B) Extrafield activity at choice point and at position below. C) High
extrafield firing at cue point before agent pauses at top of stem. Figure 10:
Place fields of four LSTM units, starting from i = 0 where the network has
been pre-trained on the sensory prediction task, drifting forwards towards
reward locations throughout reward training (where i is the number of training
iterations). The place fields ultimately rest at maze reward locations at the
end of reward training (i = 1400).


# Quantum stochastic thermodynamics: A semiclassical theory in phase space

Zhaoyu Fei<EMAIL_ADDRESS>Department of Physics and Key Laboratory of
Optical Field Manipulation of Zhejiang Province, Zhejiang Sci-Tech University,
Hangzhou 310018, China  Graduate School of China Academy of Engineering
Physics, No. 10 Xibeiwang East Road, Haidian District, Beijing, 100193, China

###### Abstract

A formalism for quantum many-body systems is proposed through a semiclassical
treatment in phase space, allowing us to establish stochastic thermodynamics
incorporating quantum statistics. Specifically, we utilize a stochastic
Fokker-Planck equation as the dynamics at the mesoscopic level. Here, the
noise term characterizing the fluctuation of the flux density accounts for the
finite-$N$ effects arising from random collisions between the system and the
reservoir. Accordingly, the stationary solution is a quasi-equilibrium state
in a canonical system. We define stochastic thermodynamic quantities based on
the trajectories of the phase-space distribution. The conservation law of
energy, $H$ theorem and fluctuation theorems are therefore obtained. Our work
sets an alternative formalism of quantum stochastic thermodynamics that is
independent of the two-point measurement scheme. The numerous projective
measurements of quantum systems are replaced by the sampling of the phase-
space distribution, offering hope for experimental verifications in the
future.

Introduction.—Stochastic thermodynamics uses stochastic variables to better
understand the non-equilibrium dynamics present in microscopic systems. In
these systems, thermal fluctuations are significant and the laws of
thermodynamics need to be understood from probabilistic perspectives. Typical
results include the definition of stochastic thermodynamic quantities and
fluctuation theorems Sekimoto2010 ; Seifert2012 ; Klages2013 ; Crooks1999 ,
the latter of which quantify the statistical behavior of nonequilibrium
systems and generalize the second law. In addition, exact thermodynamic
statements beyond the realm of linear response are obtained. These exact
results refer to distribution functions of thermodynamic quantities such as
exchanged heat, applied work, and entropy production for these systems
Seifert2012 .

Stochastic thermodynamics is also applied to quantum systems by usually
introducing the two-point measurement scheme aq2000 . Here, work is not an
observable Talkner1999 , but is defined as the energy difference between the
initial and final projective measurements. By applying the two-point
measurement scheme to open quantum systems, stochastic thermodynamics is also
established for Lindblad-type systems Hekking2013  and Caldeira-Leggett-type
systems Funo2018 . For a comprehensive overview of quantum stochastic
thermodynamics, readers are referred to Ref. Strasberg2021 .

The two-point measurement scheme assumes numerous projective measurements of
quantum systems. However, such an assumption is impractical for quantum many-
body systems or the reservoir in realistic experiments, presenting a
significant challenge to this scheme. Therefore, it is necessary to establish
quantum stochastic thermodynamics within alternative formalisms.

In our view, the reason for the conundrum is that too much information is
involved in projective measurements to establish quantum stochastic
thermodynamics. We circumvent this issue by semiclassically treating quantum
systems in phase space. When the largest energy-level spacing of the system is
small compared to the thermal excitation energy, a phase-space description of
the system under a proper error is possible through a distribution function
$\rho(z),z=(x,p)$ (a one-dimensional system is considered here for simplicity)
foot1 . Meanwhile, the temperature is low enough that the system still
presents quantum statistics, i. e., the thermal wavelength of a particle is
comparable to the interparticle spacing Pathria2011 .

In our formalism, the system is governed by a stochastic Fokker-Planck
equation that is a nonlinear equation incorporating a noise term that
characterizes the fluctuation of the flux density. The nonlinear equation
determines an equilibrium state satisfying non-Boltzmann statistics,
especially for quantum systems Kaniadakis2001 ; Frank2005 ; Chavanisa2008 . A
simple derivation using heuristic arguments is provided in Ref. Kadanoff2000 .
As an approximation of quantum Boltzmann equations, such a nonlinear equation
has been used to study finite fermionic or bosonic systems Wolschin1982 ;
Wolschin2018 , and the electron collisions in dense plasmas Daligault2016  .
The noise term keeps track of the finite-$N$ effect arising from random
collisions between the system and the reservoir. Consequently, the system
implies a quasi-equilibrium state in a canonical system, which originates from
the reverse form of the Boltzmann entropy, i. e., the exponential of the
entropy denotes the number of the corresponding microscopic states according
to Einstein’s interpretation Mishin2015 ; Einstein1910 .

In this paper, we establish the stochastic thermodynamics based on the
trajectories of the phase-space distribution. Accordingly, the stochastic
thermodynamic quantities are defined and reproduce their counterparts in
previous studies for systems consisting of distinguishable noninteracting
particles. The conservation law of energy, the $H$ theorem, and fluctuation
theorems are therefore obtained. A connection between stochastic
thermodynamics and the quasi-equilibrium state is also established.

For experimental verifications of our formalism, we would like to point out
that the current experimental techniques do not support the simultaneous
sampling of the phase-space distribution (but for the possible separate
measurements of the density distribution with high-resolution optics Bakr2009
or the momentum distribution using the time-of-flight image Ketterle1999 ).
Our formalism is hopefully verified in experiments once appropriate techniques
become available. It is emphasized that, although formulated in phase space
for the purpose of a straightforward connection to the Wigner function, our
formalism can be extended to various other spaces, where the current
experimental techniques may be sufficient.

Nonlinear Fokker-Planck equation incorporating quantum statistics.—Let us
first introduce a nonlinear Fokker-Planck equation Kaniadakis2001 ; Frank2005
; Chavanisa2008 ,

$\frac{\partial\rho_{t}}{\partial t}=L_{\text{st}}\rho_{t}+\frac{\partial
j_{t}}{\partial p},$ (1)

with a flux density in phase space $j_{t}$ originating from collisions between
the system and the reservoir. It reads

$j_{t}=\gamma p\rho_{t}(1+\epsilon\rho_{t})+\gamma
mk_{\text{B}}T\frac{\partial\rho_{t}}{\partial p},$ (2)

where $L_{\text{st}}=-\frac{p}{m}\frac{\partial}{\partial x}+\frac{\partial
U}{\partial x}\frac{\partial}{\partial p}$ denotes the streaming operator, $m$
the mass of the particle, $\gamma$ the damping coefficient, $U$ the potential
energy of the particle, $k_{\text{B}}$ the Boltzmann constant, and $T$ the
temperature of the reservoir. Here, $\epsilon=1,-1,0$ for bosons, fermions and
distinguishable particles, respectively. In the derivation, we have used the
Kramers–Moyal expansion of the master equation and truncated the expansion up
to the second order Kaniadakis2001 ; Risken1996 .

For bosons, Eq. (1) describes the evolution of particles above the critical
temperature of the Bose-Einstein condensation. The Bose-Einstein condensation
is not considered in this paper. For fermions, due to the Pauli exclusion
principle, we require $0\leq\rho_{t}\leq 1$ foot3 .

Equation (1) is conservative in particle numbers,
$N[\rho_{t}]=\int\mathrm{d}z\rho_{t}(z)$ where
$\mathrm{d}z=\mathrm{d}x\mathrm{d}p/h$, and $h$ denotes the Planck constant.
Also, it determines a steady state (a semiclassical equilibrium state in phase
space)

$\rho_{\text{eq}}(z)=\frac{1}{\mathrm{e}^{\beta\left[p^{2}/(2m)+U(x)-\mu\right]}-\epsilon},$
(3)

where $\beta=1/(k_{\text{B}}T)$ is the inverse temperature, and $\mu$ the
chemical potential. Equation (3) can be found in Refs. Giorgini1999 ;
Zaremba1999 ; Nikuni1999  for bosons by ignoring the two-body interaction, and
in Ref. Nikiforov2005  for fermions.

To highlight the connection between Eq. (1) and thermodynamics, we rewrite it
in the following form (also see Ref. Chavanisa2008 ),

$\frac{\partial\rho_{t}}{\partial
t}=L_{\text{st}}\rho_{t}+L_{\text{fp}}(\rho_{t})\frac{\delta
F[\rho_{t}]}{\delta\rho_{t}},$ (4)

where the operator $L_{\text{fp}}(\rho)=m\gamma\frac{\partial}{\partial
p}\left[\rho(1+\epsilon\rho)\frac{\partial}{\partial p}\right]$. Here for the
system, $F[\rho_{t}]=E[\rho_{t}]-TS[\rho_{t}]$ is the free energy, where the
internal energy $E[\rho_{t}]$ and the entropy $S[\rho_{t}]$ are respectively
given by

$E[\rho_{t}]=\int\mathrm{d}z\rho_{t}\left(\frac{p^{2}}{2m}+U\right),$ (5)

and

$S[\rho_{t}]=k_{\text{B}}\int\mathrm{d}z[-\rho_{t}\ln\rho_{t}+\epsilon^{-1}(1+\epsilon\rho_{t})\ln(1+\epsilon\rho_{t})].$
(6)

Stochastic Fokker-Planck equation.— It seems straightforward to construct a
Langevin equation for a single particle corresponding to the nonlinear Fokker-
Planck equation (1) (see examples in Sec. 6. 5. 4 of Ref. Frank2005 ), and
establish stochastic thermodynamics on that. However, it is inconsistent to
incorporate quantum statistics in a Langevin equation for a single particle.
This is because that quantum statistics is an effect of the exchange symmetry
of a many-body system. In contrast, we introduce fluctuations of the many-body
system in another manner: adding a noise term in the flux density $j_{t}$ (Eq.
2).

Due to the discreteness of particle numbers and the randomness of collisions
between the system and the reservoir, Eq. (1) only describes the evolution of
$\rho_{t}$ on average for a finite-size system. Hence, we add a noise term
$\eta_{t}$ into $j_{t}$ to characterize the finite-$N$ effects of the
dynamics. The introduction of the noise term in the position-space flux
density has been used to study many systems, such as many-body Brownian motion
Dean1996 , and hydrodynamic fluctuations Landau2013 . In particular, it is
used to describe the turbulence in randomly stirred fluids Forster1976 , the
onset of hydrodynamic stabilities Hohenberg1992 , and charge transport in
semiconducting devices Gu2018 . Moreover, the connection between the
stochastic Fokker-Planck equation (Eq. 7) and the Bogoliubov-Born-Green-
Kirkwood-Yvon hierarchy is discussed in Ref. Chavanis20082 .

According to the above discussions, we postulate a stochastic Fokker-Planck
equation (similar results for density in position space are shown in Refs.
Chavanis2011 ; Chavanis2019 ):

$\frac{\partial\rho^{N}_{t}}{\partial
t}=L_{\text{st}}\rho_{t}^{N}+L_{\text{fp}}(\rho_{t}^{N})\frac{\delta
F[\rho^{N}_{t}]}{\delta\rho^{N}_{t}}+\frac{\partial\eta_{t}}{\partial p},$ (7)

where $\eta_{t}$ is a Gaussian white noise satisfying
$\langle\eta_{t}(z)\rangle=0$,
$\langle\eta_{t}(z)\eta_{t^{\prime}}(z^{\prime})\rangle=2hm\gamma
k_{\text{B}}T\rho^{N}_{t}(z)[1+\epsilon\rho^{N}_{t}(z)]\delta(z-z^{\prime})\delta(t-t^{\prime})$.
Such an equation is still conservative in particle number. In the
thermodynamic limit, the suppression of the noise $\eta_{t}$ is shown in Refs.
Chavanis2011 ; Chavanis2019 .

Since the phase-space distribution $\rho_{t}^{N}$ now is a random variable, we
define the probability distribution of observing a fixed distribution $\phi$
at time $t$ as
$P_{t}[\phi]=\langle\prod_{z}\delta[\phi(z)-\rho^{N}_{t}(z)]\rangle$. Then,
following the similar procedure in Refs. Chavanis2011 ; Chavanis2019 ;
Frusawa2000 ; Goldenfeld1992 , its evolution equation (the functional Fokker-
Planck equation) is given by

$\frac{\partial P_{t}[\phi]}{\partial
t}=-\int\mathrm{d}z\frac{\delta}{\delta\phi(z)}\left\\{P_{t}[\phi]L_{\text{st}}\phi(z)+L_{\text{fp}}(\phi(z))\left[\frac{\delta
F[\phi]}{\delta\phi(z)}P_{t}[\phi]+k_{\text{B}}T\frac{\delta
P_{t}[\phi]}{\delta\phi(z)}\right]\right\\}.$ (8)

Its stationary solution is

$P_{\text{s}}[\phi]=\mathcal{Z}^{-1}\mathrm{e}^{-\beta
F[\phi]}\delta(N-N[\phi])\delta(L_{\text{st}}\phi),$ (9)

where $\mathcal{Z}=\int\mathcal{D}\phi\mathrm{e}^{-\beta
F[\phi]}\delta(N-N[\phi])\delta(L_{\text{st}}\phi)$ is the generalized
partition function. Here, the condition $L_{\text{st}}\phi=0$ means that
$\phi$ is a constant along the equienergy surface in the phase space. Also,
the integral $\int\mathcal{D}\phi$ is constrained by the condition:
$\int\mathrm{d}z\phi(z)=1$, $\phi\geq 0$ ($0\leq\phi\leq 1$ for fermions). The
stationary solution is actually a quasi-equilibrium state in a canonical
system according to the theory of equilibrium fluctuations Mishin2015 , which
precisely determines the form of the noise term Chavanis2019 . In the
thermodynamic limit, $\phi$ converges to $\rho_{\text{eq}}$ in probability by
using the method of steepest descent. In other words, according to the minimum
free-energy principle in a canonical ensemble, $P_{\text{s}}[\phi]$ now is
replaced by its most probable distribution $\rho_{\text{eq}}$.

The functional Fokker-Planck equation has an equivalent path-integral form.
Let us define a trajectory of the stochastic phase-space distribution as
$\phi_{[0,t]}:=\\{\phi_{s}|s\in[0,t]\\}$. Then, the probability distribution
of the trajectory conditioned with a fixed distribution $\phi_{0}$ at initial
time $t=0$ reads
$P[\phi_{[0,t]}|\phi_{0}]\propto\mathrm{e}^{-\beta\mathcal{S}[\phi_{[0,t]}]}$,
where the action $\mathcal{S}$ is a generalized Onsager-Machlup functional,

$\displaystyle\begin{split}\mathcal{S}[\phi_{[0,t]}]=-\frac{1}{4}\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\left[\frac{\partial\phi_{s}}{\partial
s}-L_{\text{st}}\phi_{s}-L_{\text{fp}}(\phi_{s})\frac{\delta
F[\phi_{s}]}{\delta\phi_{s}}\right]L_{\text{fp}}(\phi_{s})^{-1}\left[\frac{\partial\phi_{s}}{\partial
s}-L_{\text{st}}\phi_{s}-L_{\text{fp}}(\phi_{s})\frac{\delta
F[\phi_{s}]}{\delta\phi_{s}}\right].\end{split}$ (10)

Similar results can be found in Refs. Chavanis2011 ; Chavanis2019 ; Tauber1992
; Onsager1953 . Here, the expression $L_{\text{fp}}(\phi)^{-1}$ should be
understood as the Green’s function of $L_{\text{fp}}(\phi)$. According to the
principle of least action (the steepest descent approximation of
$\mathcal{S}[\phi_{[0,t]}]$), the dynamics of $\phi_{t}$ converges to the
nonlinear Fokker-Planck equation (4) in probability in the thermodynamic limit
foot2 .

Stochastic thermodynamics.—When the potential energy is tuned by a time-
dependent parameter $\lambda_{t}$, we can establish the stochastic
thermodynamics incorporating quantum statistics along a given trajectory of
the phase-space distribution $\phi_{[0,t]}$.

The stochastic work $w[\phi_{[0,t]}]$, and the stochastic heat
$q[\phi_{[0,t]}]$ are defined as

$\displaystyle\begin{split}w[\phi_{[0,t]}]=&\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\phi_{s}(z)\frac{\mathrm{d}\lambda_{s}}{\mathrm{d}s}\frac{\partial
U(x,\lambda_{s})}{\partial\lambda_{s}},\end{split}$ (11)

and

$\displaystyle\begin{split}q[\phi_{[0,t]}]=&\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\frac{\partial\phi_{s}(z)}{\partial
s}\left[\frac{p^{2}}{2m}+U(x,\lambda_{s})\right].\end{split}$ (12)

They satisfy the conservation law of energy

$E[\phi_{t};\lambda_{t}]-E[\phi_{0};\lambda_{0}]=w[\phi_{[0,t]}]+q[\phi_{[0,t]}].$
(13)

It is worth mentioning that for a deterministic trajectory of $\phi_{s}$, the
definitions of work (11) and heat (12) are the semiclassical phase-space
counterparts of the quantum work and the quantum heat given in Ref. quan2007 .

The stochastic entropy $s[\phi_{t}]$, stochastic total entropy production
$s_{\text{p}}[\phi_{[0,t]}]$, and stochastic free energy,
$f[\phi_{t};\lambda_{t}]$ are respectively given by

$\displaystyle\begin{split}s[\phi_{t}]=S[\phi_{t}]-k_{\text{B}}\ln
P_{t}[\phi_{t}],\end{split}$ (14)
$s_{\text{p}}[\phi_{[0,t]}]=s[\phi_{t}]-s[\phi_{0}]+s_{\text{r}}[\phi_{[0,t]}],$
(15)
$f[\phi_{t};\lambda_{t}]=E[\phi_{t};\lambda_{t}]-Ts[\phi_{t}]=F[\phi_{t};\lambda_{t}]+k_{\text{B}}T\ln
P_{t}[\phi_{t}],$ (16)

where $P_{t}[\phi]$ is the solution of the functional Fokker-Planck equation
(8), and $s_{\text{r}}[\phi_{[0,t]}]=-q[\phi_{[0,t]}]/T$ denotes the entropy
change of the reservoir.

While the stochastic internal energy $E[\phi_{t};\lambda_{t}]$, the stochastic
work $w[\phi_{[0,t]}]$, and the stochastic heat $q[\phi_{[0,t]}]$ are solely
determined by $\phi_{t}$, Eq. (14) means that the stochastic entropy
$s[\phi_{t}]$ also depends on the probability distribution of the stochastic
phase-space distribution $P_{t}[\phi]$. That is to say, the explicit
expression of $s[\phi_{t}]$ cannot be determined until we have obtained the
solution of the functional Fokker-Planck equation. Moreover, if we take an
average of $s[\phi_{t}]$ over $P_{t}[\phi]$, we have

$\langle s[\phi_{t}]\rangle=\langle S[\phi_{t}]\rangle-
k_{\text{B}}\int\mathcal{D}\phi P_{t}[\phi]\ln P_{t}[\phi].$ (17)

The term $-k_{\text{B}}\ln P_{t}[\phi_{t}]$ actually has a contribution to the
average entropy due to the probability distribution $P_{t}[\phi]$ from the
perspective of information theory Jaynes1957 ; Jaynes1983 . Also, this term is
essential to prove the $H$ theorem $\frac{\mathrm{d}}{\mathrm{d}t}\langle
s_{\text{p}}[\phi_{[0,t]}]\rangle\geq 0$ (see Appendix A).

We prove the fluctuation theorems according to these stochastic thermodynamic
quantities. For this purpose, let $P[\phi_{[0,t]}|\phi_{0}]$
($P^{\dagger}[\phi^{{\dagger}}_{[0,t]}|\phi^{{\dagger}}_{0}]$) denote the
conditional probability distribution of the forward (reverse) trajectory
$\phi_{[0,t]}$
($\phi_{[0,t]}^{{\dagger}}:=\\{\phi^{\dagger}_{s}|s\in[0,t]\\},\phi^{\dagger}_{s}(x,p):=\phi_{t-s}(x,-p)$)
with a fixed initial distribution $\phi_{0}$ ($\phi^{{\dagger}}_{0}$) and a
forward (reverse) protocol $\lambda_{s}$
($\lambda^{\dagger}_{s}:=\lambda_{t-s}$). Then, using the generalized Onsager-
Machlup functional (Eq. 10), we obtain the detailed fluctuation theorems (see
Aappendix C):

$\displaystyle\begin{split}\ln\frac{P[\phi_{[0,t]}|\phi_{0}]}{P^{\dagger}[\phi^{{\dagger}}_{[0,t]}|\phi^{{\dagger}}_{0}]}=\ln\frac{P_{t}[\phi_{t}]}{P_{0}[\phi_{0}]}+\frac{s_{\text{p}}[\phi_{[0,t]}]}{k_{\text{B}}}.\end{split}$
(18)

By adding an arbitrary normalized distribution of $\phi$ at the initial time
of the forward (reverse) process $P_{0}[\phi_{0}]$
($P^{\prime}_{0}[\phi_{0}^{\dagger}]$), we obtain the integral fluctuation
theorems:

$\left\langle\frac{P^{\prime}_{0}[\phi_{0}^{\dagger}]}{P_{t}[\phi_{t}]}\mathrm{e}^{-s_{\text{p}}/k_{\text{B}}}\right\rangle=1.$
(19)

Such an equality is formally consistent with the integral fluctuation theorems
in previous studies Seifert2012 ; Klages2013 .

For a choice of $P^{\prime}_{0}[\phi^{\dagger}_{0}]=P_{t}[\phi_{t}]$, we
obtain the integral fluctuation theorem for total entropy production
Seifert2012 ; Klages2013 ,

$\left\langle\mathrm{e}^{-s_{\text{p}}/k_{\text{B}}}\right\rangle=1.$ (20)

Then as a corollary, the second law $\left\langle
s_{\text{p}}\right\rangle\geq 0$ follows from the fluctuation theorem by using
Jensen’s inequality.

Moreover, when both $P_{0}[\phi_{0}],P^{\prime}_{0}[\phi_{0}^{\dagger}]$ are
stationary solutions of the functional Fokker-Planck equation (Eq. 9), i. e.,
$P_{0}[\phi_{0}]=P_{\text{s}}[\phi_{0};\lambda_{0}],P^{\prime}_{0}[\phi_{0}^{\dagger}]=P_{\text{s}}[\phi_{t};\lambda_{t}]$,
we obtain the generalized Jarzynski equality

$\left\langle\mathrm{e}^{-\beta
w}\right\rangle=\frac{\mathcal{Z}(\lambda_{t})}{\mathcal{Z}(\lambda_{0})},$
(21)

and the generalized principle of maximum work

$\langle w\rangle\geq-
k_{\text{B}}T\ln\left[\frac{\mathcal{Z}(\lambda_{t})}{\mathcal{Z}(\lambda_{0})}\right]$
(22)

by using Jensen’s inequality. In the limit
$N\to\infty,\mathcal{Z}(\lambda_{t})/\mathcal{Z}(\lambda_{0})=\mathrm{e}^{-\beta(F[\rho_{\text{eq}}(\lambda_{t})]-F[\rho_{\text{eq}}(\lambda_{0})])}$
by using the steepest descent approximation of $\mathcal{Z}$, which further
results in the Jarzynski equality $\langle\mathrm{e}^{-\beta
w}\rangle=\mathrm{e}^{-\beta(F[\rho_{\text{eq}}(\lambda_{t})]-F[\rho_{\text{eq}}(\lambda_{0})])}$,
and the principle of maximum work $\langle w\rangle\geq
F[\rho_{\text{eq}}(\lambda_{t})]-F[\rho_{\text{eq}}(\lambda_{0})]$
Sekimoto2010 ; Seifert2012 ; Klages2013 .

We would like to emphasize that for the stationary solution
$P_{\text{s}}[\phi;\lambda]$ (9), we have

$\langle f[\phi;\lambda]\rangle=-k_{\text{B}}T\ln\mathcal{Z}(\lambda),$ (23)
$\langle
E[\phi;\lambda]\rangle=-\frac{\partial\ln\mathcal{Z}(\lambda)}{\partial\beta},$
(24) $\frac{\langle
s[\phi;\lambda]\rangle}{k_{\text{B}}}=\ln\mathcal{Z}(\lambda)-\frac{\partial\ln\mathcal{Z}(\lambda)}{\partial\beta}.$
(25)

Hence, the generalized partition function $\mathcal{Z}(\lambda)$ as the
characteristic state function of a canonical quasi-equilibrium state plays the
same role as the partition function of a canonical equilibrium state.

For distinguishable particles ($\epsilon=0$), our formalism coincides with
previous stochastic thermodynamics. It is demonstrated by following Dean’s
study of a system of $N$-particle Langevin equations Dean1996 ,

$\displaystyle\begin{split}\frac{\mathrm{d}x_{i}(t)}{\mathrm{d}t}=&\frac{p_{i}(t)}{m},\\\
\frac{\mathrm{d}p_{i}(t)}{\mathrm{d}t}=&-\frac{\partial U(x_{i}(t))}{\partial
x_{i}(t)}-\gamma p_{i}(t)+\xi_{i}(t),\end{split}$ (26)

where the white noises are uncorrelated, $\langle\xi_{i}(t)\rangle=0$,
$\langle\xi_{i}(t)\xi_{j}(t^{\prime})\rangle=2m\gamma
k_{\text{B}}T\delta_{ij}\delta(t-t^{\prime})$. Then, the empirical density
$\tilde{\rho}^{N}_{t}(z)=\sum_{i=1}^{N}\delta(z-z_{i}(t))$ of particles
satisfies the stochastic Fokker-Planck equation (7), where
$\eta_{t}=\sum_{i=1}^{N}\xi_{i}(t)$ correspondingly. Substituting
$\tilde{\rho}^{N}_{t}(z)$ into the definition of stochastic thermodynamic
quantities $E[\phi_{t};\lambda_{t}]$, $w[\phi_{[0,t]}]$, $q[\phi_{[0,t]}]$,
and $s[\phi_{t}]$ (the second term in Eq. (14)), we reproduce their
counterparts in previous studies of stochastic thermodynamics Sekimoto2010 ;
Seifert2012 ; Klages2013 .

Conclusion.—For a quantum many-body system semiclassically treated by a phase-
space distribution at the mesoscopic level, we consider a stochastic Fokker-
Planck equation as the dynamics of the system at the mesoscopic level. Here,
the noise term finds its origin in the discreteness of the particle number and
the randomness of collisions between the system and the reservoir, embodying
Einstein’s interpretation of the reverse form of the Boltzmann entropy.

Based on trajectories of the stochastic phase-space distribution, we propose a
formalism of stochastic thermodynamics that accounts for quantum statistics.
Consequently, a connection between stochastic thermodynamics and the quasi-
equilibrium state is established.

Independent of the previous formalism that relies on the two-point measurement
scheme, our formalism is based on the sampling of the stochastic phase-space
distribution, offering hope for experimental verifications. Moreover, by
incorporating mean-field interactions and considering other types of non-
Boltzmann entropy Kaniadakis2001 ; Frank2005 ; Chavanisa2008 , our formalism
can be readily extended to interacting systems and other non-Boltzmann
systems. It is worth mentioning that an efficient treatment of fractional
exclusion statistics is given by using the nonlinear Fokker-Planck equation in
Ref. Kaniadakis1996 .

The author thanks Y. Chen, C. P. Sun, J. F. Chen, and J. Gu for helpful
suggestions. This work is supported by the NSFC (Grant No. 12088101), NSAF
(Grants No. U1930403, and No. U1930402), and the China Postdoctoral Science
Foundation (No. 2021M700359).

## Appendix A: proof of the $H$ theorem

It follows from Eqs. (15, 16) and the conservation of energy that

$\displaystyle\begin{split}&\frac{\mathrm{d}}{\mathrm{d}t}\langle
s_{\text{p}}[\phi_{[0,t]}]\rangle\\\
=&\frac{1}{T}\frac{\mathrm{d}}{\mathrm{d}t}\left(\langle
w[\phi_{[0,t]}]\rangle-\langle f[\phi_{t};\lambda_{t}]\rangle\right)\\\
=&\frac{1}{T}\frac{\mathrm{d}}{\mathrm{d}t}\left(\langle
w[\phi_{[0,t]}]\rangle-\langle F[\phi_{t};\lambda_{t}]+k_{\text{B}}T\ln
P_{t}[\phi_{t}]\rangle\right).\end{split}$ (A.1)

Substituting Eq. (11) into Eq. (LABEL:e26), and using Eq. (8) we have
(noticing $\frac{\partial
F[\phi;\lambda_{t}]}{\partial\lambda_{t}}=\int\mathrm{d}z\phi(z)\frac{\mathrm{d}\lambda_{t}}{\mathrm{d}t}\frac{\partial
U(x,\lambda_{t})}{\partial\lambda_{t}}$)

$\displaystyle\begin{split}&\frac{1}{T}\frac{\mathrm{d}}{\mathrm{d}t}\left(\langle
w[\phi_{[0,t]}]\rangle-\langle F[\phi_{t};\lambda_{t}]+k_{\text{B}}T\ln
P_{t}[\phi_{t}]\rangle\right)\\\
=&\frac{1}{T}\left(\int\mathrm{d}z\langle\phi_{t}(z)\rangle\frac{\mathrm{d}\lambda_{t}}{\mathrm{d}t}\frac{\partial
U(x,\lambda_{t})}{\partial\lambda_{t}}-\frac{\mathrm{d}}{\mathrm{d}t}\langle
F[\phi_{t};\lambda_{t}]+k_{\text{B}}T\ln P_{t}[\phi_{t}]\rangle\right)\\\
=&\frac{1}{T}\left(\int\mathcal{D}\phi
P_{t}[\phi]\int\mathrm{d}z\phi(z)\frac{\mathrm{d}\lambda_{t}}{\mathrm{d}t}\frac{\partial
U(x,\lambda_{t})}{\partial\lambda_{t}}-\frac{\mathrm{d}}{\mathrm{d}t}\langle
F[\phi_{t};\lambda_{t}]+k_{\text{B}}T\ln P_{t}[\phi_{t}]\rangle\right)\\\
=&\frac{1}{T}\left(\int\mathcal{D}\phi
P_{t}[\phi]\frac{\mathrm{d}\lambda_{t}}{\mathrm{d}t}\frac{\partial
F[\phi;\lambda_{t}]}{\partial\lambda_{t}}-\frac{\mathrm{d}}{\mathrm{d}t}\int\mathcal{D}\phi
P_{t}[\phi](F[\phi;\lambda_{t}]+k_{\text{B}}T\ln P_{t}[\phi])\right)\\\
=&-\int\mathcal{D}\phi\frac{\partial P_{t}[\phi]}{\partial
t}(T^{-1}F[\phi;\lambda_{t}]+k_{\text{B}}\ln P_{t}[\phi]+k_{\text{B}})\\\
=&\int\mathcal{D}\phi\int\mathrm{d}z\frac{\delta}{\delta\phi(z)}\left\\{P_{t}[\phi]L_{\text{st}}\phi(z)+L_{\text{fp}}(\phi(z))\left[\frac{\delta
F[\phi;\lambda_{t}]}{\delta\phi(z)}P_{t}[\phi]+k_{\text{B}}T\frac{\delta
P_{t}[\phi]}{\delta\phi(z)}\right]\right\\}(T^{-1}F[\phi;\lambda_{t}]+k_{\text{B}}\ln
P_{t}[\phi]+k_{\text{B}}).\end{split}$ (A.2)

Finally, using integral by parts, we obtain

$\displaystyle\begin{split}&\int\mathcal{D}\phi\int\mathrm{d}z\frac{\delta}{\delta\phi(z)}\left\\{P_{t}[\phi]L_{\text{st}}\phi(z)+L_{\text{fp}}(\phi(z))\left[\frac{\delta
F[\phi;\lambda_{t}]}{\delta\phi(z)}P_{t}[\phi]+k_{\text{B}}T\frac{\delta
P_{t}[\phi]}{\delta\phi(z)}\right]\right\\}(T^{-1}F[\phi;\lambda_{t}]+k_{\text{B}}\ln
P_{t}[\phi]+k_{\text{B}})\\\ =&\int\mathcal{D}\phi
P_{t}[\phi]\int\mathrm{d}z\left\\{-\frac{1}{T}\frac{\delta
F[\phi;\lambda_{t}]}{\delta\phi(z)}+k_{\text{B}}\frac{\delta}{\delta\phi(z)}\right\\}L_{\text{st}}\phi(z)\\\
&+\int\mathcal{D}\phi\int\mathrm{d}z\frac{\gamma\phi(z)[1+\epsilon\phi(z)]}{TP_{t}[\phi]}\left\\{\frac{\partial}{\partial
p}\left[\frac{\delta
F[\phi;\lambda_{t}]}{\delta\phi(z)}P_{t}[\phi]+k_{\text{B}}T\frac{\delta
P_{t}[\phi]}{\delta\phi(z)}\right]\right\\}^{2}.\end{split}$ (A.3)

Because the first term actually equals 0 by using the two identities in
Appendix B and the second term is not less than 0, we obtain the $H$ theorem
$\frac{\mathrm{d}}{\mathrm{d}t}\langle s_{\text{p}}[\phi_{[0,t]}]\rangle\geq
0$.

## Appendix B: two identities in Eq. (LABEL:eh)

For the first identity, using $F[\phi]=E[\phi]-TS[\phi]$ and integral by
parts, we have

$\displaystyle\begin{split}&\int\mathrm{d}z\frac{\delta
F[\phi]}{\delta\phi(z)}L_{\text{st}}\phi(z)\\\
=&\int\mathrm{d}z\left[\frac{p^{2}}{2m}+U(x,\lambda)\right]L_{\text{st}}\phi(z)-T\int\mathrm{d}z\frac{\delta
S[\phi]}{\delta\phi(z)}L_{\text{st}}\phi(z)\\\ =&-\int\mathrm{d}z\phi
L_{\text{st}}\left[\frac{p^{2}}{2m}+U(x,\lambda)\right]-k_{\text{B}}T\int\mathrm{d}zL_{\text{st}}[-\phi\ln\phi+\epsilon^{-1}(1+\epsilon\phi)\ln(1+\epsilon\phi)]\\\
=&0.\end{split}$ (B.1)

For the second identity, let us first define a set of real orthonormal
complete square-integrable functions $\\{u_{i}(z)\\}$ in the phase space.
Then, we decompose $\phi(z)$ as

$\phi(z)=\sum_{i}\phi_{i}u_{i}(z),$ (B.2)

where

$\phi_{i}=\int dz\phi(z)u_{i}(z).$ (B.3)

Hence, it follows from Eq. (B.3) that

$\displaystyle\begin{split}\int\mathrm{d}z\frac{\delta}{\delta\phi(z)}L_{\text{st}}\phi(z)=&\sum_{i}\int\mathrm{d}z\frac{\delta\phi_{i}}{\delta\phi(z)}L_{\text{st}}u_{i}(z)\\\
=&\sum_{i}\int\mathrm{d}zu_{i}(z)L_{\text{st}}u_{i}(z)\\\
=&-\sum_{i}\int\mathrm{d}zu_{i}(z)L_{\text{st}}u_{i}(z)\\\ =&0,\end{split}$
(B.4)

The minus sign in the third equality results from integral by parts.

## Appendix C: proof of the detailed fluctuation theorem

We first prove that the normalization factor of the forward propagator
$P[\phi_{[0,t]}|\phi_{0}]$ equals that of the reverse propagator
$P^{\dagger}[\phi^{{\dagger}}_{[0,t]}|\phi^{{\dagger}}_{0}]$. Let us define
another white noise $\zeta_{t}=\frac{\partial}{\partial p}\eta_{t}$, where
$\langle\zeta_{t}(z)\rangle=0$ and

$\displaystyle\begin{split}\langle\zeta_{t}(z)\zeta_{t^{\prime}}(z^{\prime})\rangle=&\frac{\partial^{2}}{\partial
p\partial p^{\prime}}\langle\eta_{t}(z)\eta_{t^{\prime}}(z^{\prime})\rangle\\\
=&2hk_{\text{B}}T\frac{\partial}{\partial
p}\left\\{\rho^{N}_{t}(z)[1+\epsilon\rho^{N}_{t}(z)]\frac{\partial}{\partial
p^{\prime}}\delta(z-z^{\prime})\right\\}\delta(t-t^{\prime})\\\
=&-2hk_{\text{B}}T\frac{\partial}{\partial
p}\left\\{\rho^{N}_{t}(z)[1+\epsilon\rho^{N}_{t}(z)]\frac{\partial}{\partial
p}\delta(z-z^{\prime})\right\\}\delta(t-t^{\prime})\\\
=&-2hk_{\text{B}}TL_{\text{fp}}(\phi_{t}(z))\delta(z-z^{\prime})\delta(t-t^{\prime}).\end{split}$
(C.1)

Thus, the probability distribution of a trajectory $\zeta_{[0,t]}$ reads

$\displaystyle\begin{split}P[\zeta_{[0,t]}]=\mathcal{N}[\phi_{[0,t]}]\mathrm{e}^{\frac{\beta}{4}\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\zeta_{s}(z)L_{\text{fp}}(\phi_{s})^{-1}\zeta_{s}(z)},\end{split}$
(C.2)

where

$\mathcal{N}[\phi_{[0,t]}]=\int\mathcal{D}\zeta_{[0,t]}\mathrm{e}^{\frac{\beta}{4}\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\zeta_{s}(z)L_{\text{fp}}(\phi_{s})^{-1}\zeta_{s}(z)}.$
(C.3)

We obtain the propagator $P[\phi_{[0,t]}|\phi_{0}]$ through

$\displaystyle\begin{split}&P[\phi_{[0,t]}|\phi_{0}]\\\
=&\int\mathcal{D}\zeta_{[0,t]}P[\zeta_{[0,t]}]\prod_{(s,z)}\delta\left(\frac{\partial\phi_{s}(z)}{\partial
s}-L_{\text{st}}\phi_{s}(z)-L_{\text{fp}}(\phi_{s}(z))\frac{\delta
F[\phi_{s}]}{\delta\phi_{s}(z)}-\zeta_{s}(z)\right)\\\
=&\mathcal{J}[\phi_{[0,t]}]\mathcal{N}[\phi_{[0,t]}]\mathrm{e}^{-\beta\mathcal{S}[\phi_{[0,t]}]},\end{split}$
(C.4)

where, the Jacobian $\mathcal{J}[\phi_{[0,t]}]$ stemming from the variable
transformation from $\zeta$ to $\phi$ is just a constant when the standard Ito
forward discretization is applied Tauber1992 .

According to Eq. (C.3) and $\phi^{\dagger}_{s}(x,p)=\phi_{t-s}(x,-p)$, we have
the following equality:

$\displaystyle\begin{split}\mathcal{N}[\phi^{\dagger}_{[0,t]}]=&\int\mathcal{D}\zeta_{[0,t]}\mathrm{e}^{\frac{\beta}{4}\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\zeta_{s}(z)L_{\text{fp}}(\phi^{\dagger}_{s})^{-1}\zeta_{s}(z)}\\\
=&\int\mathcal{D}\zeta_{[0,t]}\mathrm{e}^{\frac{\beta}{4}\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\zeta_{s}(z)L_{\text{fp}}(\phi_{s})^{-1}\zeta_{s}(z)}\\\
=&\mathcal{N}[\phi_{[0,t]}],\end{split}$ (C.5)

where we have used the transformation $s\to
t-s,p\to-p,\zeta_{t-s}(x,-p)\to\zeta_{s}(x,p)$.

Next, using the transformation $s\to t-s,p\to-p$, it is straightforward to
verify that the action $\mathcal{S}[\phi^{\dagger}_{[0,t]}]$ (Eq. (10)) can be
rewritten in terms of $\phi_{[0,t]}$:

$\mathcal{S}[\phi^{\dagger}_{[0,t]}]=-\frac{1}{4}\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\left[\frac{\partial\phi_{s}}{\partial
s}-L_{\text{st}}\phi_{s}+L_{\text{fp}}(\phi_{s})\frac{\delta
F[\phi_{s}]}{\delta\phi_{s}}\right]L_{\text{fp}}(\phi_{s})^{-1}\left[\frac{\partial\phi_{s}}{\partial
s}-L_{\text{st}}\phi_{s}+L_{\text{fp}}(\phi_{s})\frac{\delta
F[\phi_{s}]}{\delta\phi_{s}}\right].$ (C.6)

Finally, we derive the detailed fluctuation theorem:

$\displaystyle\begin{split}\ln\frac{P[\phi_{[0,t]}|\phi_{0}]}{P^{\dagger}[\phi^{{\dagger}}_{[0,t]}|\phi^{{\dagger}}_{0}]}=&-\beta(\mathcal{S}[\phi_{[0,t]}]-\mathcal{S}[\phi^{\dagger}_{[0,t]}])\\\
=&-\beta\int_{0}^{t}\mathrm{d}s\int\mathrm{d}z\frac{\partial\phi_{s}}{\partial
s}\frac{\delta F[\phi_{s};\lambda_{s}]}{\delta\phi_{s}}\\\
=&-\beta(F[\phi_{t};\lambda_{t}]-F[\phi_{0};\lambda_{0}]-w[\phi_{[0,t]}])\\\
=&\ln\frac{P_{t}[\phi_{t}]}{P_{0}[\phi_{0}]}+\frac{s_{\text{p}}[\phi_{[0,t]}]}{k_{\text{B}}},\end{split}$
(C.7)

where in the first line we have used Eq. (C.5), in the second line we have
used Eqs. (10, C.6), in the third line we have used
$\frac{\mathrm{d}\lambda_{t}}{\mathrm{d}t}\frac{\partial
F[\phi_{t};\lambda_{t}]}{\partial\lambda_{t}}=\frac{\mathrm{d}\lambda_{t}}{\mathrm{d}t}\frac{\partial
E[\phi_{t};\lambda_{t}]}{\partial\lambda_{t}}=\frac{\mathrm{d}}{\mathrm{d}t}w[\phi_{[0,t]}]$,
and in the fourth line we have used the definitions of the stochastic
thermodynamic quantities (Eqs. (11-16)).

## References

  * (1) K. Sekimoto, _Stochastic energetics_ (Springer, Berlin, 2010).
  * (2) U. Seifert, Reports on progress in physics 75(12), 126001 (2012).
  * (3) _Nonequilibrium statistical physics of small systems_ , edited by R. Klages, W. Just, and C. Jarzynski (Wiley-VCH, Weinheim, Ger., 2013).
  * (4) G. E. Crooks, Phys. Rev. E 60(3), 2721 (1999); G. Hummer, and A. Szabo, Proc. Natl. Acad. Sci. U. S. A. 98(7), 3658-3661 (2001).
  * (5) J. Kurchan, arXiv: cond-mat/0007360; H. Tasaki, arXiv: cond-mat/0009244.
  * (6) P. Talkner, E. Lutz and P. Hänggi, Phys. Rev. E 75, 050102(R) (2007).
  * (7) F. W. J. Hekking and J. P. Pekola, Phys. Rev. Lett. 111, 093602 (2013); M. Esposito, U. Harbola, and S. Mukamel, Rev. Mod. Phys. 81, 1665 (2009); J. M. Horowitz and J. M. Parrondo, New J. Phys. 15, 085028 (2013); F. Liu, Prog. Phys. 38, 1 (2018).
  * (8) K. Funo, H. T. Quan, Phys. Rev. Lett. 121, 040602 (2018); Phys. Rev. E 98, 012113 (2018).
  * (9) P. Strasberg, _Quantum Stochastic Thermodynamics: Foundations and Selected Applications_ (Oxford University Press, Oxford, U. K., 2022).
  * (10) The phase-space distribution is defined by using the Wigner function of the one-particle density matrix [see Eqs. (17, 18) in Ref. Zaremba1999 ].
  * (11) R. K. Pathria, and P. D. Beale, _Statistical Mechanics_ , 3rd ed. (Academic Press, Cambridge, MA, 2011).
  * (12) G. Kaniadakis, Physica A 296, 405–425 (2001); Physics Letters A 288, 283–291 (2001).
  * (13) T. D. Frank, _Nonlinear Fokker–Planck Equations: Fundamentals and Applications_ ( Springer, Berlin, 2005).
  * (14) P. H. Chavanisa, Eur. Phys. J. B 62, 179–208 (2008).
  * (15) L. P. Kadanoff, _Statistical physics: statics, dynamics and renormalization_ (World Scientific, Singapore, 2000).
  * (16) G. Wolschin, Phys. Rev. Lett. 48(15), 1004 (1982).
  * (17) G. Wolschin, Physica A 499, 1-10 (2018).
  * (18) P. Danielewicz, Physica A 100(1), 167-182 (1980); J. Daligault, Physics of Plasmas, 23(3), 032706 (2016).
  * (19) Y. Mishin, Annals of Physics 363, 48–97 (2015).
  * (20) A. Einstein, Ann. Phys., Lpz. 33 1275–1298 (1910).
  * (21) W. S. Bakr, J. I. Gillen, A. Peng, S. Föling, and M. Greiner, Nature 462, 74-77 (2009); L. W. Cheuk, M. A. Nichols, M. Okan, T. Gersdorf, V. V. Ramasesh, W. S. Bakr, T. Lompe, and M. W. Zwierlein, Phys. Rev. Lett. 114, 193001 (2015).
  * (22) W. Ketterle, D. S. Durfee, and D. M. Stamper-Kurn, in _Bose-Einstein condensation in atomic gases_ , Proceedings of the International School of Physics “Enrico Fermi”, Course CXL, Varenna, 1998, edited by M. Inguscio, S. Stringari, and C. E. Wieman (IOS Press, Amsterdam, 1999), p. 67.
  * (23) H. Risken, _The Fokker-Planck Equation_ (Springer, Berlin, 1996).
  * (24) With a proper initial distribution $0\leq\rho_{0}\leq 1$, the phase-space density $\rho_{t}$ remains bounded by $0\leq\rho_{t}\leq 1$ during the whole evolution. It is proved by following the similar procedure in Ref. Risken1996  for $\rho_{t}\geq 0$ and in Ref. Carrillo2008  for $\rho_{t}\leq 1$.
  * (25) J. A. Carrillo, J. Rosado, F. Salvarani, Appl. Math. Lett. 21, 148–154 (2008).
  * (26) E. Zaremba, T. Nikuni, and A. Griffin, J. Low Temp. Phys. 116, 277 (1999).
  * (27) S. Giorgini, L. P. Pitaevskii, and S. Stringari, J. Low Temp. Phys. 109, 309 (1997).
  * (28) T. Nikuni, E. Zaremba, and A. Griffin, Phys. Rev. Lett. 83, 10 (1999).
  * (29) A. F. Nikiforov,V. G. Novikov, V. B. Uvarov, _Quantum-Statistical Models of Hot Dense Matter: Methods for Computation Opacity and Equation of State_ (Springer, Berlin, 2005).
  * (30) D. S. Dean, J. Phys. A Math. Gen. 29, L613–L617, (1996).
  * (31) L. D. Landau, E. M. Lifshitz, _Fluid Mechanics_ , Course of Theoretical Physics, Vol. 6 (Elsevier, Amsterdam, 2013).
  * (32) D. Forster, D. R. Nelson and M. J. Stephen, Phys. Rev. Lett. 36, 867 (1976); Phys. Rev. A 16, 732 (1977).
  * (33) P. C. Hohenberg and J. B. Swift, Phys. Rev. A 46, 4773 (1992); J. B. Swift, K. L. Babcock and P. C. Hohenberg, Physica A 204, 625 (1994).
  * (34) J. Gu, and P. Gaspard, Phys. Rev. E 97, 052138 (2018); 99, 012137 (2019).
  * (35) P. H. Chavanis, Physica A 387, 5716–5740 (2008).
  * (36) P. H. Chavanis, Physica A 390, 1546–1574 (2011); Entropy 17, 3205 (2015); P. H. Chavanis, and L. Delfini, Phys. Rev. E 89, 032139 (2014).
  * (37) P. H. Chavanis, Entropy 21, 1006 (2019).
  * (38) H. Frusawa, and R. Hayakawa, J. Phys. A: Math. Gen. 33, L155–L160 (2000).
  * (39) N. Goldenfeld, _Lectures on Phase Transitions and the Renormalization Group_ ( Addison-Wesley, Reading, MA, 1992).
  * (40) U. C. Täuber , _Critical dynamics: a field theory approach to equilibrium and nonequilibrium scaling behavior_ (Cambridge University Press, Cambridge, U. K., 2014).
  * (41) L. Onsager, S. Machlup, Phys. Rev. 91, 1505–1512 (1953).
  * (42) This conclusion is a result of the scalings of the following quantities in the thermodynamic limit: $\rho\sim\phi\sim p\sim N^{0},x\sim t\sim N$.
  * (43) H. T. Quan, Y. X. Liu, C. P. Sun, and F. Nori, Phys. Rev. E 76, 031105 (2007).
  * (44) E. T. Jaynes, _Information theory and statistical mechanics, I and II_. Phys. Rev. 1957, 106, 620.
  * (45) _E. T. Jaynes: Papers on Probability, Statistics and Statistical Physics_ , edited by R. D. Rosenkrantz, (Reidel, Dordrecht, 1983).
  * (46) G. Kaniadakis, A. Lavagno, P. Quarati, Nucl. Phys. B 466, 527 (1996).